Gradual decoding refresh in video coding

ABSTRACT

A method of decoding a coded video bitstream implemented by a video decoder is provided. The method includes the video decoder determining whether a gradual decoding refresh (GDR) picture in the coded video bitstream is an initial picture in decoding order or the initial picture following an end of sequence (EOS) network abstraction layer (NAL) unit in the decoding order; setting a flag to a value to prevent the GDR picture from being output when the GDR picture is the initial picture in decoding order or the initial picture following the EOS NAL unit in the decoding order; decoding the GDR picture; and storing the GDR picture in a decoded picture buffer (DPB).

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is a continuation of International ApplicationNo. PCT/US2020/022182, filed Mar. 11, 2020 by Ye-Kui Wang, et. al., andtitled “Gradual Decoding Refresh In Video Coding,” which claims thebenefit of U.S. Provisional Patent Application No. 62/816,722, filedMar. 11, 2019, by Fnu Hendry, et al., and titled “Gradual DecodingRefresh in Video Coding,” and U.S. Provisional Patent Application No.62/871,020, filed Jul. 5, 2019, by Fnu Hendry, et al., and titled“Gradual Decoding Refresh in Video Coding,” each of which is herebyincorporated by reference.

TECHNICAL FIELD

In general, this disclosure describes techniques supporting gradualdecoding refresh in video coding. More specifically, this disclosureallows progressive intra refresh to enable random access without havingto use intra random access point (IRAP) pictures.

BACKGROUND

The amount of video data needed to depict even a relatively short videocan be substantial, which may result in difficulties when the data is tobe streamed or otherwise communicated across a communications networkwith limited bandwidth capacity. Thus, video data is generallycompressed before being communicated across modern daytelecommunications networks. The size of a video could also be an issuewhen the video is stored on a storage device because memory resourcesmay be limited. Video compression devices often use software and/orhardware at the source to code the video data prior to transmission orstorage, thereby decreasing the quantity of data needed to representdigital video images. The compressed data is then received at thedestination by a video decompression device that decodes the video data.With limited network resources and ever increasing demands of highervideo quality, improved compression and decompression techniques thatimprove compression ratio with little to no sacrifice in image qualityare desirable.

SUMMARY

A first aspect relates to a method of decoding a coded video bitstreamimplemented by a video decoder. The method includes determining, by thevideo decoder, whether a gradual decoding refresh (GDR) picture in thecoded video bitstream is an initial picture in decoding order or is theinitial picture following an end of sequence (EOS) network abstractionlayer (NAL) unit in the decoding order; setting, by the video decoder, aflag to a value to prevent the GDR picture from being output when theGDR picture is the initial picture in decoding order or the initialpicture following the EOS NAL unit in the decoding order; decoding, bythe video decoder, the GDR picture; and storing, by the video decoder,the GDR picture in a decoded picture buffer (DPB).

The method provides techniques that allow progressive intra refresh toenable random access without having to use an intra random access point(TRAP) picture. The value of a flag is set to prevent a GDR picture frombeing output when the GDR picture is the initial picture in decodingorder or the initial picture following the EOS NAL unit in the decodingorder. Setting the flag in this manner prevents potentially dirty datafrom being output to a display. That is, the value of the flag settingcontrols whether potentially dirty data from the GDR picture is outputor whether the video decoder will wait for full synchronization to begindisplaying data. By having the ability to restrict dirty data from beingoutput, the coder/decoder (a.k.a., “codec”) in video coding is improvedrelative to current codecs. As a practical matter, the improved videocoding process offers the user a better user experience when videos aresent, received, and/or viewed.

In a first implementation form of the method according to the firstaspect as such, the method further comprises setting the flag to thevalue to prevent the GDR picture and any trailing pictures between theGDR picture and the recovery point picture in output order from beingoutput when the GDR picture is the initial picture in decoding order orthe initial picture following the EOS NAL unit in the decoding order.

In a second implementation form of the method according to the firstaspect as such or any preceding implementation form of the first aspect,the GDR picture is the initial picture in a coded video sequence (CVS)of the coded video bitstream, or the GDR picture is the initial picturein a layer of the coded video bitstream.

In a third implementation form of the method according to the firstaspect as such or any preceding implementation form of the first aspect,the layer is a coded layer video sequence (CLVS) of a coded videosequence (CVS) of the coded video bitstream.

In a fourth implementation form of the method according to the firstaspect as such or any preceding implementation form of the first aspect,the value of the flag is set to one (1) to prevent the GDR picture andany trailing pictures between the GDR picture and a recovery pointpicture in output order from being output when the GDR picture is theinitial picture in the decoding order or the initial picture followingthe EOS NAL unit in the decoding order.

In a fifth implementation form of the method according to the firstaspect as such or any preceding implementation form of the first aspect,the value of the flag is set to zero (0) when the GDR picture is not theinitial picture in the decoding order and is not the initial picturefollowing the EOS NAL unit in the decoding order.

A second aspect relates to a decoding device. The decoding deviceincludes a receiver configured to receive a coded video bitstream; amemory coupled to the receiver, the memory storing instructions; and aprocessor coupled to the memory, the processor configured to execute theinstructions to cause the decoding device to: determine whether agradual decoding refresh (GDR) picture in the coded video bitstream isan initial picture in decoding order or is the initial picture followingan end of sequence (EOS) network abstraction layer (NAL) unit in thedecoding order; set a flag to a value to prevent the GDR picture frombeing output when the GDR picture is the initial picture in decodingorder or the initial picture following the EOS NAL unit in the decodingorder; decode the GDR picture; and store the GDR picture in a decodedpicture buffer (DPB).

The decoding device provides techniques that allow progressive intrarefresh to enable random access without having to use an intra randomaccess point (TRAP) picture. The value of a flag is set to prevent a GDRpicture from being output when the GDR picture is the initial picture indecoding order or the initial picture following the EOS NAL unit in thedecoding order. Setting the flag in this manner prevents potentiallydirty data from being output to a display. That is, the value of theflag setting controls whether potentially dirty data from the GDRpicture is output or whether the video decoder will wait for fullsynchronization to begin displaying data. By having the ability torestrict dirty data from being output, the coder/decoder (a.k.a.,“codec”) in video coding is improved relative to current codecs. As apractical matter, the improved video coding process offers the user abetter user experience when videos are sent, received, and/or viewed.

In a first implementation form of the decoding device according to thesecond aspect as such, the flag is set to the value to prevent the GDRpicture and any trailing pictures between the GDR picture and therecovery point picture in output order from being output when the GDRpicture is the initial picture in decoding order or the initial picturefollowing the EOS NAL unit in the decoding order.

In a second implementation form of the decoding device according to thesecond aspect as such, the GDR picture is the initial picture in a codedvideo sequence (CVS) of the coded video bitstream, or the GDR picture isthe initial picture in a layer of the coded video bitstream.

In a third implementation form of the decoding device according to thesecond aspect as such, the layer is a coded layer video sequence (CLVS)of a coded video sequence (CVS) of the coded video bitstream.

In a fourth implementation form of the decoding device according to thesecond aspect as such, the value of the flag is set to one (1) toprevent the GDR picture and any trailing pictures between the GDRpicture and a recovery point picture in output order from being outputwhen the GDR picture is the initial picture in the decoding order or theinitial picture following the EOS NAL unit in the decoding order.

In a fifth implementation form of the decoding device according to thesecond aspect as such, the value of the flag is set to zero (0) when theGDR picture is not the initial picture in the decoding order and is notthe initial picture following the EOS NAL unit in the decoding order.

A third aspect relates to a coding apparatus. The coding apparatusincludes a receiver configured to receive a picture to encode or toreceive a bitstream to decode; a transmitter coupled to the receiver,the transmitter configured to transmit the bitstream to a decoder or totransmit a decoded image to a display; a memory coupled to at least oneof the receiver or the transmitter, the memory configured to storeinstructions; and a processor coupled to the memory, the processorconfigured to execute the instructions stored in the memory to performany of the methods disclosed herein.

The coding apparatus provides techniques that allow progressive intrarefresh to enable random access without having to use an intra randomaccess point (TRAP) picture. The value of a flag is set to prevent a GDRpicture from being output when the GDR picture is the initial picture indecoding order or the initial picture following the EOS NAL unit in thedecoding order. Setting the flag in this manner prevents potentiallydirty data from being output to a display. That is, the value of theflag setting controls whether potentially dirty data from the GDRpicture is output or whether the video decoder will wait for fullsynchronization to begin displaying data. By having the ability torestrict dirty data from being output, the coder/decoder (a.k.a.,“codec”) in video coding is improved relative to current codecs. As apractical matter, the improved video coding process offers the user abetter user experience when videos are sent, received, and/or viewed.

In a first implementation form of the coding apparatus according to thethird aspect as such, further comprising a display configured to displayan image.

A fourth aspect relates to a system. The system includes an encoder; anda decoder in communication with the encoder, wherein the encoder or thedecoder includes the decoding device, the encoding device, or the codingapparatus disclosed herein.

The system provides techniques that allow progressive intra refresh toenable random access without having to use an intra random access point(TRAP) picture. The value of a flag is set to prevent a GDR picture frombeing output when the GDR picture is the initial picture in decodingorder or the initial picture following the EOS NAL unit in the decodingorder. Setting the flag in this manner prevents potentially dirty datafrom being output to a display. That is, the value of the flag settingcontrols whether potentially dirty data from the GDR picture is outputor whether the video decoder will wait for full synchronization to begindisplaying data. By having the ability to restrict dirty data from beingoutput, the coder/decoder (a.k.a., “codec”) in video coding is improvedrelative to current codecs. As a practical matter, the improved videocoding process offers the user a better user experience when videos aresent, received, and/or viewed.

A fifth aspect relates to a means for coding. The means for codingincludes receiving means configured to receive a picture to encode or toreceive a bitstream to decode; transmission means coupled to thereceiving means, the transmission means configured to transmit thebitstream to a decoding means or to transmit a decoded image to adisplay means; storage means coupled to at least one of the receivingmeans or the transmission means, the storage means configured to storeinstructions; and processing means coupled to the storage means, theprocessing means configured to execute the instructions stored in thestorage means to perform any of the methods disclosed herein.

The means for coding provides techniques that allow progressive intrarefresh to enable random access without having to use an intra randomaccess point (TRAP) picture. The value of a flag is set to prevent a GDRpicture from being output when the GDR picture is the initial picture indecoding order or the initial picture following the EOS NAL unit in thedecoding order. Setting the flag in this manner prevents potentiallydirty data from being output to a display. That is, the value of theflag setting controls whether potentially dirty data from the GDRpicture is output or whether the video decoder will wait for fullsynchronization to begin displaying data. By having the ability torestrict dirty data from being output, the coder/decoder (a.k.a.,“codec”) in video coding is improved relative to current codecs. As apractical matter, the improved video coding process offers the user abetter user experience when videos are sent, received, and/or viewed.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of this disclosure, reference is nowmade to the following brief description, taken in connection with theaccompanying drawings and detailed description, wherein like referencenumerals represent like parts.

FIG. 1 is a block diagram illustrating an example coding system that mayutilize GDR techniques.

FIG. 2 is a block diagram illustrating an example video encoder that mayimplement GDR techniques.

FIG. 3 is a block diagram illustrating an example of a video decoderthat may implement GDR techniques.

FIG. 4 is a representation of a relationship between an IRAP picturerelative to leading pictures and trailing pictures in a decoding orderand a presentation order.

FIG. 5 illustrates a gradual decoding refresh technique.

FIG. 6 is a schematic diagram illustrating an undesirable motion search.

FIG. 7 illustrates a video bitstream configured to implement a gradualdecoding refresh technique according to an embodiment of the disclosure.

FIG. 8 is an embodiment of a method of decoding a coded video bitstream.

FIG. 9 is a schematic diagram of a video coding device.

FIG. 10 is a schematic diagram of an embodiment of a means for coding.

DETAILED DESCRIPTION

FIG. 1 is a block diagram illustrating an example coding system 10 thatmay utilize video coding techniques as described herein. As shown inFIG. 1 , the coding system 10 includes a source device 12 that providesencoded video data to be decoded at a later time by a destination device14. In particular, the source device 12 may provide the video data todestination device 14 via a computer-readable medium 16. Source device12 and destination device 14 may comprise any of a wide range ofdevices, including desktop computers, notebook (e.g., laptop) computers,tablet computers, set-top boxes, telephone handsets such as so-called“smart” phones, so-called “smart” pads, televisions, cameras, displaydevices, digital media players, video gaming consoles, video streamingdevice, or the like. In some cases, source device 12 and destinationdevice 14 may be equipped for wireless communication.

Destination device 14 may receive the encoded video data to be decodedvia computer-readable medium 16. Computer-readable medium 16 maycomprise any type of medium or device capable of moving the encodedvideo data from source device 12 to destination device 14. In oneexample, computer-readable medium 16 may comprise a communication mediumto enable source device 12 to transmit encoded video data directly todestination device 14 in real-time. The encoded video data may bemodulated according to a communication standard, such as a wirelesscommunication protocol, and transmitted to destination device 14. Thecommunication medium may comprise any wireless or wired communicationmedium, such as a radio frequency (RF) spectrum or one or more physicaltransmission lines. The communication medium may form part of apacket-based network, such as a local area network, a wide-area network,or a global network such as the Internet. The communication medium mayinclude routers, switches, base stations, or any other equipment thatmay be useful to facilitate communication from source device 12 todestination device 14.

In some examples, encoded data may be output from output interface 22 toa storage device. Similarly, encoded data may be accessed from thestorage device by input interface. The storage device may include any ofa variety of distributed or locally accessed data storage media such asa hard drive, Blu-ray discs, digital video disks (DVD)s, Compact DiscRead-Only Memories (CD-ROMs), flash memory, volatile or non-volatilememory, or any other suitable digital storage media for storing encodedvideo data. In a further example, the storage device may correspond to afile server or another intermediate storage device that may store theencoded video generated by source device 12. Destination device 14 mayaccess stored video data from the storage device via streaming ordownload. The file server may be any type of server capable of storingencoded video data and transmitting that encoded video data to thedestination device 14. Example file servers include a web server (e.g.,for a website), a file transfer protocol (FTP) server, network attachedstorage (NAS) devices, or a local disk drive. Destination device 14 mayaccess the encoded video data through any standard data connection,including an Internet connection. This may include a wireless channel(e.g., a Wi-Fi connection), a wired connection (e.g., digital subscriberline (DSL), cable modem, etc.), or a combination of both that issuitable for accessing encoded video data stored on a file server. Thetransmission of encoded video data from the storage device may be astreaming transmission, a download transmission, or a combinationthereof.

The techniques of this disclosure are not necessarily limited towireless applications or settings. The techniques may be applied tovideo coding in support of any of a variety of multimedia applications,such as over-the-air television broadcasts, cable televisiontransmissions, satellite television transmissions, Internet streamingvideo transmissions, such as dynamic adaptive streaming over HTTP(DASH), digital video that is encoded onto a data storage medium,decoding of digital video stored on a data storage medium, or otherapplications. In some examples, coding system 10 may be configured tosupport one-way or two-way video transmission to support applicationssuch as video streaming, video playback, video broadcasting, and/orvideo telephony.

In the example of FIG. 1 , source device 12 includes video source 18,video encoder 20, and output interface 22. Destination device 14includes input interface 28, video decoder 30, and display device 32. Inaccordance with this disclosure, video encoder 20 of the source device12 and/or the video decoder 30 of the destination device 14 may beconfigured to apply the techniques for video coding. In other examples,a source device and a destination device may include other components orarrangements. For example, source device 12 may receive video data froman external video source, such as an external camera. Likewise,destination device 14 may interface with an external display device,rather than including an integrated display device.

The illustrated coding system 10 of FIG. 1 is merely one example.Techniques for video coding may be performed by any digital videoencoding and/or decoding device. Although the techniques of thisdisclosure generally are performed by a video coding device, thetechniques may also be performed by a video encoder/decoder, typicallyreferred to as a “CODEC.” Moreover, the techniques of this disclosuremay also be performed by a video preprocessor. The video encoder and/orthe decoder may be a graphics processing unit (GPU) or a similar device.

Source device 12 and destination device 14 are merely examples of suchcoding devices in which source device 12 generates coded video data fortransmission to destination device 14. In some examples, source device12 and destination device 14 may operate in a substantially symmetricalmanner such that each of the source and destination devices 12, 14includes video encoding and decoding components. Hence, coding system 10may support one-way or two-way video transmission between video devices12, 14, e.g., for video streaming, video playback, video broadcasting,or video telephony.

Video source 18 of source device 12 may include a video capture device,such as a video camera, a video archive containing previously capturedvideo, and/or a video feed interface to receive video from a videocontent provider. As a further alternative, video source 18 may generatecomputer graphics-based data as the source video, or a combination oflive video, archived video, and computer-generated video.

In some cases, when video source 18 is a video camera, source device 12and destination device 14 may form so-called camera phones or videophones. As mentioned above, however, the techniques described in thisdisclosure may be applicable to video coding in general, and may beapplied to wireless and/or wired applications. In each case, thecaptured, pre-captured, or computer-generated video may be encoded byvideo encoder 20. The encoded video information may then be output byoutput interface 22 onto a computer-readable medium 16.

Computer-readable medium 16 may include transient media, such as awireless broadcast or wired network transmission, or storage media (thatis, non-transitory storage media), such as a hard disk, flash drive,compact disc, digital video disc, Blu-ray disc, or othercomputer-readable media. In some examples, a network server (not shown)may receive encoded video data from source device 12 and provide theencoded video data to destination device 14, e.g., via networktransmission. Similarly, a computing device of a medium productionfacility, such as a disc stamping facility, may receive encoded videodata from source device 12 and produce a disc containing the encodedvideo data. Therefore, computer-readable medium 16 may be understood toinclude one or more computer-readable media of various forms, in variousexamples.

Input interface 28 of destination device 14 receives information fromcomputer-readable medium 16. The information of computer-readable medium16 may include syntax information defined by video encoder 20, which isalso used by video decoder 30, that includes syntax elements thatdescribe characteristics and/or processing of blocks and other codedunits, e.g., group of pictures (GOPs). Display device 32 displays thedecoded video data to a user, and may comprise any of a variety ofdisplay devices such as a cathode ray tube (CRT), a liquid crystaldisplay (LCD), a plasma display, an organic light emitting diode (OLED)display, or another type of display device.

Video encoder 20 and video decoder 30 may operate according to a videocoding standard, such as the High Efficiency Video Coding (HEVC)standard presently under development, and may conform to the HEVC TestModel (HM). Alternatively, video encoder 20 and video decoder 30 mayoperate according to other proprietary or industry standards, such asthe International Telecommunications Union TelecommunicationStandardization Sector (ITU-T) H.264 standard, alternatively referred toas Moving Picture Expert Group (MPEG)-4, Part 10, Advanced Video Coding(AVC), H.265/HEVC, or extensions of such standards. The techniques ofthis disclosure, however, are not limited to any particular codingstandard. Other examples of video coding standards include MPEG-2 andITU-T H.263. Although not shown in FIG. 1 , in some aspects, videoencoder 20 and video decoder 30 may each be integrated with an audioencoder and decoder, and may include appropriatemultiplexer-demultiplexer (MUX-DEMUX) units, or other hardware andsoftware, to handle encoding of both audio and video in a common datastream or separate data streams. If applicable, MUX-DEMUX units mayconform to the ITU H.223 multiplexer protocol, or other protocols suchas the user datagram protocol (UDP).

Video encoder 20 and video decoder 30 each may be implemented as any ofa variety of suitable encoder circuitry, such as one or moremicroprocessors, digital signal processors (DSPs), application specificintegrated circuits (ASICs), field programmable gate arrays (FPGAs),discrete logic, software, hardware, firmware or any combinationsthereof. When the techniques are implemented partially in software, adevice may store instructions for the software in a suitable,non-transitory computer-readable medium and execute the instructions inhardware using one or more processors to perform the techniques of thisdisclosure. Each of video encoder 20 and video decoder 30 may beincluded in one or more encoders or decoders, either of which may beintegrated as part of a combined encoder/decoder (CODEC) in a respectivedevice. A device including video encoder 20 and/or video decoder 30 maycomprise an integrated circuit, a microprocessor, and/or a wirelesscommunication device, such as a cellular telephone.

FIG. 2 is a block diagram illustrating an example of video encoder 20that may implement video coding techniques. Video encoder 20 may performintra- and inter-coding of video blocks within video slices.Intra-coding relies on spatial prediction to reduce or remove spatialredundancy in video within a given video frame or picture. Inter-codingrelies on temporal prediction to reduce or remove temporal redundancy invideo within adjacent frames or pictures of a video sequence. Intra-mode(I mode) may refer to any of several spatial based coding modes.Inter-modes, such as uni-directional (a.k.a., uni prediction) prediction(P mode) or bi-prediction (a.k.a., bi prediction) (B mode), may refer toany of several temporal-based coding modes.

As shown in FIG. 2 , video encoder 20 receives a current video blockwithin a video frame to be encoded. In the example of FIG. 2 , videoencoder 20 includes mode select unit 40, reference frame memory 64,summer 50, transform processing unit 52, quantization unit 54, andentropy coding unit 56. Mode select unit 40, in turn, includes motioncompensation unit 44, motion estimation unit 42, intra-prediction(a.k.a., intra prediction) unit 46, and partition unit 48. For videoblock reconstruction, video encoder 20 also includes inversequantization unit 58, inverse transform unit 60, and summer 62. Adeblocking filter (not shown in FIG. 2 ) may also be included to filterblock boundaries to remove blockiness artifacts from reconstructedvideo. If desired, the deblocking filter would typically filter theoutput of summer 62. Additional filters (in loop or post loop) may alsobe used in addition to the deblocking filter. Such filters are not shownfor brevity, but if desired, may filter the output of summer 50 (as anin-loop filter).

During the encoding process, video encoder 20 receives a video frame orslice to be coded. The frame or slice may be divided into multiple videoblocks. Motion estimation unit 42 and motion compensation unit 44perform inter-predictive coding of the received video block relative toone or more blocks in one or more reference frames to provide temporalprediction. Intra-prediction unit 46 may alternatively performintra-predictive coding of the received video block relative to one ormore neighboring blocks in the same frame or slice as the block to becoded to provide spatial prediction. Video encoder 20 may performmultiple coding passes, e.g., to select an appropriate coding mode foreach block of video data.

Moreover, partition unit 48 may partition blocks of video data intosub-blocks, based on evaluation of previous partitioning schemes inprevious coding passes. For example, partition unit 48 may initiallypartition a frame or slice into largest coding units (LCUs), andpartition each of the LCUs into sub-coding units (sub-CUs) based onrate-distortion analysis (e.g., rate-distortion optimization). Modeselect unit 40 may further produce a quad-tree data structure indicativeof partitioning of a LCU into sub-CUs. Leaf-node CUs of the quad-treemay include one or more prediction units (PUs) and one or more transformunits (TUs).

The present disclosure uses the term “block” to refer to any of a CU,PU, or TU, in the context of HEVC, or similar data structures in thecontext of other standards (e.g., macroblocks and sub-blocks thereof inH.264/AVC). A CU includes a coding node, PUs, and TUs associated withthe coding node. A size of the CU corresponds to a size of the codingnode and is square in shape. The size of the CU may range from 8×8pixels up to the size of the treeblock with a maximum of 64×64 pixels orgreater. Each CU may contain one or more PUs and one or more TUs. Syntaxdata associated with a CU may describe, for example, partitioning of theCU into one or more PUs. Partitioning modes may differ between whetherthe CU is skip or direct mode encoded, intra-prediction mode encoded, orinter-prediction (a.k.a., inter prediction) mode encoded. PUs may bepartitioned to be non-square in shape. Syntax data associated with a CUmay also describe, for example, partitioning of the CU into one or moreTUs according to a quad-tree. A TU can be square or non-square (e.g.,rectangular) in shape.

Mode select unit 40 may select one of the coding modes, intra- orinter-, e.g., based on error results, and provides the resulting intra-or inter-coded block to summer 50 to generate residual block data and tosummer 62 to reconstruct the encoded block for use as a reference frame.Mode select unit 40 also provides syntax elements, such as motionvectors, intra-mode indicators, partition information, and other suchsyntax information, to entropy coding unit 56.

Motion estimation unit 42 and motion compensation unit 44 may be highlyintegrated, but are illustrated separately for conceptual purposes.Motion estimation, performed by motion estimation unit 42, is theprocess of generating motion vectors, which estimate motion for videoblocks. A motion vector, for example, may indicate the displacement of aPU of a video block within a current video frame or picture relative toa predictive block within a reference frame (or other coded unit)relative to the current block being coded within the current frame (orother coded unit). A predictive block is a block that is found toclosely match the block to be coded, in terms of pixel difference, whichmay be determined by sum of absolute difference (SAD), sum of squaredifference (SSD), or other difference metrics. In some examples, videoencoder 20 may calculate values for sub-integer pixel positions ofreference pictures stored in reference frame memory 64. For example,video encoder 20 may interpolate values of one-quarter pixel positions,one-eighth pixel positions, or other fractional pixel positions of thereference picture. Therefore, motion estimation unit 42 may perform amotion search relative to the full pixel positions and fractional pixelpositions and output a motion vector with fractional pixel precision.

Motion estimation unit 42 calculates a motion vector for a PU of a videoblock in an inter-coded slice by comparing the position of the PU to theposition of a predictive block of a reference picture. The referencepicture may be selected from a first reference picture list (List 0) ora second reference picture list (List 1), each of which identify one ormore reference pictures stored in reference frame memory 64. Motionestimation unit 42 sends the calculated motion vector to entropyencoding unit 56 and motion compensation unit 44.

Motion compensation, performed by motion compensation unit 44, mayinvolve fetching or generating the predictive block based on the motionvector determined by motion estimation unit 42. Again, motion estimationunit 42 and motion compensation unit 44 may be functionally integrated,in some examples. Upon receiving the motion vector for the PU of thecurrent video block, motion compensation unit 44 may locate thepredictive block to which the motion vector points in one of thereference picture lists. Summer 50 forms a residual video block bysubtracting pixel values of the predictive block from the pixel valuesof the current video block being coded, forming pixel difference values,as discussed below. In general, motion estimation unit 42 performsmotion estimation relative to luma components, and motion compensationunit 44 uses motion vectors calculated based on the luma components forboth chroma components and luma components. Mode select unit 40 may alsogenerate syntax elements associated with the video blocks and the videoslice for use by video decoder 30 in decoding the video blocks of thevideo slice.

Intra-prediction unit 46 may intra-predict a current block, as analternative to the inter-prediction performed by motion estimation unit42 and motion compensation unit 44, as described above. In particular,intra-prediction unit 46 may determine an intra-prediction mode to useto encode a current block. In some examples, intra-prediction unit 46may encode a current block using various intra-prediction modes, e.g.,during separate encoding passes, and intra-prediction unit 46 (or modeselect unit 40, in some examples) may select an appropriateintra-prediction mode to use from the tested modes.

For example, intra-prediction unit 46 may calculate rate-distortionvalues using a rate-distortion analysis for the various testedintra-prediction modes, and select the intra-prediction mode having thebest rate-distortion characteristics among the tested modes.Rate-distortion analysis generally determines an amount of distortion(or error) between an encoded block and an original, unencoded blockthat was encoded to produce the encoded block, as well as a bitrate(that is, a number of bits) used to produce the encoded block.Intra-prediction unit 46 may calculate ratios from the distortions andrates for the various encoded blocks to determine which intra-predictionmode exhibits the best rate-distortion value for the block.

In addition, intra-prediction unit 46 may be configured to code depthblocks of a depth map using a depth modeling mode (DMM). Mode selectunit 40 may determine whether an available DMM mode produces bettercoding results than an intra-prediction mode and the other DMM modes,e.g., using rate-distortion optimization (RDO). Data for a texture imagecorresponding to a depth map may be stored in reference frame memory 64.Motion estimation unit 42 and motion compensation unit 44 may also beconfigured to inter-predict depth blocks of a depth map.

After selecting an intra-prediction mode for a block (e.g., aconventional intra-prediction mode or one of the DMM modes),intra-prediction unit 46 may provide information indicative of theselected intra-prediction mode for the block to entropy coding unit 56.Entropy coding unit 56 may encode the information indicating theselected intra-prediction mode. Video encoder 20 may include in thetransmitted bitstream configuration data, which may include a pluralityof intra-prediction mode index tables and a plurality of modifiedintra-prediction mode index tables (also referred to as codeword mappingtables), definitions of encoding contexts for various blocks, andindications of a most probable intra-prediction mode, anintra-prediction mode index table, and a modified intra-prediction modeindex table to use for each of the contexts.

Video encoder 20 forms a residual video block by subtracting theprediction data from mode select unit 40 from the original video blockbeing coded. Summer 50 represents the component or components thatperform this subtraction operation.

Transform processing unit 52 applies a transform, such as a discretecosine transform (DCT) or a conceptually similar transform, to theresidual block, producing a video block comprising residual transformcoefficient values. Transform processing unit 52 may perform othertransforms which are conceptually similar to DCT. Wavelet transforms,integer transforms, sub-band transforms or other types of transformscould also be used.

Transform processing unit 52 applies the transform to the residualblock, producing a block of residual transform coefficients. Thetransform may convert the residual information from a pixel value domainto a transform domain, such as a frequency domain. Transform processingunit 52 may send the resulting transform coefficients to quantizationunit 54. Quantization unit 54 quantizes the transform coefficients tofurther reduce bit rate. The quantization process may reduce the bitdepth associated with some or all of the coefficients. The degree ofquantization may be modified by adjusting a quantization parameter. Insome examples, quantization unit 54 may then perform a scan of thematrix including the quantized transform coefficients. Alternatively,entropy encoding unit 56 may perform the scan.

Following quantization, entropy coding unit 56 entropy codes thequantized transform coefficients. For example, entropy coding unit 56may perform context adaptive variable length coding (CAVLC), contextadaptive binary arithmetic coding (CABAC), syntax-based context-adaptivebinary arithmetic coding (SBAC), probability interval partitioningentropy (PIPE) coding or another entropy coding technique. In the caseof context-based entropy coding, context may be based on neighboringblocks. Following the entropy coding by entropy coding unit 56, theencoded bitstream may be transmitted to another device (e.g., videodecoder 30) or archived for later transmission or retrieval.

Inverse quantization unit 58 and inverse transform unit 60 apply inversequantization and inverse transformation, respectively, to reconstructthe residual block in the pixel domain, e.g., for later use as areference block. Motion compensation unit 44 may calculate a referenceblock by adding the residual block to a predictive block of one of theframes of reference frame memory 64. Motion compensation unit 44 mayalso apply one or more interpolation filters to the reconstructedresidual block to calculate sub-integer pixel values for use in motionestimation. Summer 62 adds the reconstructed residual block to themotion compensated prediction block produced by motion compensation unit44 to produce a reconstructed video block for storage in reference framememory 64. The reconstructed video block may be used by motionestimation unit 42 and motion compensation unit 44 as a reference blockto inter-code a block in a subsequent video frame.

FIG. 3 is a block diagram illustrating an example of video decoder 30that may implement video coding techniques. In the example of FIG. 3 ,video decoder 30 includes an entropy decoding unit 70, motioncompensation unit 72, intra-prediction unit 74, inverse quantizationunit 76, inverse transformation unit 78, reference frame memory 82, andsummer 80. Video decoder 30 may, in some examples, perform a decodingpass generally reciprocal to the encoding pass described with respect tovideo encoder 20 (FIG. 2 ). Motion compensation unit 72 may generateprediction data based on motion vectors received from entropy decodingunit 70, while intra-prediction unit 74 may generate prediction databased on intra-prediction mode indicators received from entropy decodingunit 70.

During the decoding process, video decoder 30 receives an encoded videobitstream that represents video blocks of an encoded video slice andassociated syntax elements from video encoder 20. Entropy decoding unit70 of the video decoder 30 entropy decodes the bitstream to generatequantized coefficients, motion vectors or intra-prediction modeindicators, and other syntax elements. Entropy decoding unit 70 forwardsthe motion vectors and other syntax elements to motion compensation unit72. Video decoder 30 may receive the syntax elements at the video slicelevel and/or the video block level.

When the video slice is coded as an intra-coded (I) slice,intra-prediction unit 74 may generate prediction data for a video blockof the current video slice based on a signaled intra-prediction mode anddata from previously decoded blocks of the current frame or picture.When the video frame is coded as an inter-coded (e.g., B, P, or GPB)slice, motion compensation unit 72 produces predictive blocks for avideo block of the current video slice based on the motion vectors andother syntax elements received from entropy decoding unit 70. Thepredictive blocks may be produced from one of the reference pictureswithin one of the reference picture lists. Video decoder 30 mayconstruct the reference frame lists, List 0 and List 1, using defaultconstruction techniques based on reference pictures stored in referenceframe memory 82.

Motion compensation unit 72 determines prediction information for avideo block of the current video slice by parsing the motion vectors andother syntax elements, and uses the prediction information to producethe predictive blocks for the current video block being decoded. Forexample, motion compensation unit 72 uses some of the received syntaxelements to determine a prediction mode (e.g., intra- orinter-prediction) used to code the video blocks of the video slice, aninter-prediction slice type (e.g., B slice, P slice, or GPB slice),construction information for one or more of the reference picture listsfor the slice, motion vectors for each inter-encoded video block of theslice, inter-prediction status for each inter-coded video block of theslice, and other information to decode the video blocks in the currentvideo slice.

Motion compensation unit 72 may also perform interpolation based oninterpolation filters. Motion compensation unit 72 may use interpolationfilters as used by video encoder 20 during encoding of the video blocksto calculate interpolated values for sub-integer pixels of referenceblocks. In this case, motion compensation unit 72 may determine theinterpolation filters used by video encoder 20 from the received syntaxelements and use the interpolation filters to produce predictive blocks.

Data for a texture image corresponding to a depth map may be stored inreference frame memory 82. Motion compensation unit 72 may also beconfigured to inter-predict depth blocks of a depth map.

Keeping the above in mind, video compression techniques perform spatial(intra-picture) prediction and/or temporal (inter-picture) prediction toreduce or remove redundancy inherent in video sequences. For block-basedvideo coding, a video slice (i.e., a video picture or a portion of avideo picture) may be partitioned into video blocks, which may also bereferred to as treeblocks, coding tree blocks (CTBs), coding tree units(CTUs), coding units (CUs) and/or coding nodes. Video blocks in anintra-coded (I) slice of a picture are encoded using spatial predictionwith respect to reference samples in neighboring blocks in the samepicture. Video blocks in an inter-coded (P or B) slice of a picture mayuse spatial prediction with respect to reference samples in neighboringblocks in the same picture or temporal prediction with respect toreference samples in other reference pictures. Pictures may be referredto as frames, and reference pictures may be referred to as referenceframes.

Spatial or temporal prediction results in a predictive block for a blockto be coded. Residual data represents pixel differences between theoriginal block to be coded and the predictive block. An inter-codedblock is encoded according to a motion vector that points to a block ofreference samples forming the predictive block, and the residual dataindicating the difference between the coded block and the predictiveblock. An intra-coded block is encoded according to an intra-coding modeand the residual data. For further compression, the residual data may betransformed from the pixel domain to a transform domain, resulting inresidual transform coefficients, which then may be quantized. Thequantized transform coefficients, initially arranged in atwo-dimensional array, may be scanned in order to produce aone-dimensional vector of transform coefficients, and entropy coding maybe applied to achieve even more compression.

Image and video compression has experienced rapid growth, leading tovarious coding standards. Such video coding standards include ITU-TH.261, ISO/IEC MPEG-1 Part 2, ITU-T H.262 or ISO/IEC MPEG-2 Part 2,ITU-T H.263, ISO/IEC MPEG-4 Part 2, Advanced Video Coding (AVC), alsoknown as ITU-T H.264 or ISO/IEC MPEG-4 Part 10, and High EfficiencyVideo Coding (HEVC), also known as ITU-T H.265 or MPEG-H Part 2. AVCincludes extensions such as Scalable Video Coding (SVC), Multiview VideoCoding (MVC) and Multiview Video Coding plus Depth (MVC+D), and 3D AVC(3D-AVC). HEVC includes extensions such as Scalable HEVC (SHVC),Multiview HEVC (MV-HEVC), and 3D HEVC (3D-HEVC).

There is also a new video coding standard, named Versatile Video Coding(VVC), being developed by the joint video experts team (JVET) of ITU-Tand ISO/IEC. While the VVC standard has several working drafts, oneWorking Draft (WD) of VVC in particular, namely B. Bross, J. Chen, andS. Liu, “Versatile Video Coding (Draft 4),” JVET-M1001-v5, 13th JVETMeeting, January 2019 (VVC Draft 4) is referenced herein.

The description of the techniques disclosed herein are based on theunder-development video coding standard Versatile Video Coding (VVC) bythe joint video experts team (JVET) of ITU-T and ISO/IEC. However, thetechniques also apply to other video codec specifications.

FIG. 4 is a representation 400 of a relationship between an IRAP picture402 relative to leading pictures 404 and trailing pictures 406 in adecoding order 408 and a presentation order 410. In an embodiment, theIRAP picture 402 is referred to as a clean random access (CRA) pictureor as an instantaneous decoder refresh (IDR) picture with random accessdecodable leading (RADL) picture. In HEVC, IDR pictures, CRA pictures,and Broken Link Access (BLA) pictures are all considered IRAP pictures402. For VVC, during the 12th NET meeting in October 2018, it was agreedto have both IDR and CRA pictures as IRAP pictures.

As shown in FIG. 4 , the leading pictures 404 (e.g., pictures 2 and 3)follow the IRAP picture 402 in the decoding order 408, but precede theIRAP picture 402 in the presentation order 410. The trailing picture 406follows the IRAP picture 402 in both the decoding order 408 and in thepresentation order 410. While two leading pictures 404 and one trailingpicture 406 are depicted in FIG. 4 , those skilled in the art willappreciate that more or fewer leading pictures 404 and/or trailingpictures 406 may be present in the decoding order 408 and thepresentation order 410 in practical applications.

The leading pictures 404 in FIG. 4 have been divided into two types,namely random access skipped leading (RASL) and RADL. When decodingstarts with the IRAP picture 402 (e.g., picture 1), the RADL picture(e.g., picture 3) can be properly decoded; however, the RASL picture(e.g., picture 2) cannot be properly decoded. Thus, the RASL picture isdiscarded. In light of the distinction between RADL and RASL pictures,the type of leading picture associated with the IRAP picture should beidentified as either RADL or RASL for efficient and proper coding. InHEVC, when RASL and RADL pictures are present, it is constrained thatfor RASL and RADL pictures that are associated with the same IRAPpicture, the RASL pictures shall precede the RADL pictures inpresentation order 410.

An TRAP picture 402 provides the following two importantfunctionalities/benefits. Firstly, the presence of an IRAP picture 402indicates that the decoding process can start from that picture. Thisfunctionality allows a random access feature in which the decodingprocess starts at that position in the bitstream, not necessarily thebeginning of the bitstream, as long as an TRAP picture 402 is present atthat position. Secondly, the presence of an IRAP picture 402 refreshesthe decoding process such that a coded picture starting at the IRAPpicture 402, excluding RASL pictures, are coded without any reference toprevious pictures. Having an IRAP picture 402 present in the bitstreamconsequently would stop any error that may happen during decoding ofcoded pictures prior to the TRAP picture 402 to propagate to the TRAPpicture 402 and those pictures that follow the TRAP picture 402 indecoding order 408.

While TRAP pictures 402 provide important functionalities, they comewith a penalty to the compression efficiency. The presence of an TRAPpicture 402 causes a surge in bitrate. This penalty to the compressionefficiency is due to two reasons. Firstly, as an IRAP picture 402 is anintra-predicted picture, the picture itself would require relativelymore bits to represent when compared to other pictures (e.g., leadingpictures 404, trailing pictures 406) that are inter-predicted pictures.Secondly, because the presence of an TRAP picture 402 breaks temporalprediction (this is because the decoder would refresh decoding process,in which one of the actions of the decoding process for this is toremove previous reference pictures in the decoded picture buffer (DPB)),the TRAP picture 402 causes the coding of pictures that follow the TRAPpicture 402 in decoding order 408 to be less efficient (i.e., needs morebits to represent) because they have less reference pictures for theirinter-prediction coding.

Among the picture types that are considered TRAP pictures 402, the IDRpicture in HEVC has different signaling and derivation when compared toother picture types. Some of the differences are as follows.

For signaling and derivation of a picture order count (POC) value of anIDR picture, the most significant bit (MSB) part of the POC is notderived from previous key picture but simply set to be equal to 0.

For signaling information needed for reference picture management, theslice header of an IDR picture does not contain information needed to besignaled to assist reference picture management. For other picture types(i.e., CRA, Trailing, temporal sub-layer access (TSA), etc.),information such as the reference picture set (RPS) described below orother forms of similar information (e.g., reference picture lists) areneeded for the reference pictures marking process (i.e., the process todetermine the status of reference pictures in the decoded picture buffer(DPB), either used for reference and unused for reference). However, forthe IDR picture, such information does not need to be signaled becausethe presence of IDR indicates that the decoding process shall simplymark all reference pictures in the DPB as unused for reference.

In HEVC and VVC, TRAP pictures 402 and leading pictures 404 may each becontained within a single network abstraction layer (NAL) unit. A set ofthe NAL units may be referred to as an access unit. TRAP pictures 402and leading pictures 404 are given different NAL unit types so that theycan be easily identified by system level applications. For example, avideo splicer needs to understand coded picture types without having tounderstand too much detail of the syntax element in the coded bitstream,particularly to identify TRAP pictures 402 from non-TRAP pictures and toidentify leading pictures 404, including determining RASL and RADLpictures, from trailing pictures 406. Trailing pictures 406 are thosepictures that are associated with an TRAP picture 402 and follow theTRAP picture 402 in presentation order 410. A picture may follow theparticular IRAP picture 402 in decoding order 408 and precede any otherIRAP picture 402 in decoding order 408. For this, giving TRAP pictures402 and leading pictures 404 their own NAL unit type helps suchapplications.

For HEVC, NAL unit types for TRAP pictures include the following:

-   -   BLA with leading picture (BLA_W_LP): NAL unit of a Broken Link        Access (BLA) picture that may be followed by one or more leading        pictures in decoding order.    -   BLA with RADL (BLA_W_RADL): NAL unit of a BLA picture that may        be followed by one or more RADL pictures but no RASL picture in        decoding order.    -   BLA with no leading picture (BLA_N_LP): NAL unit of a BLA        picture that is not followed by leading picture in decoding        order.    -   IDR with RADL (IDR_W_RADL): NAL unit of an IDR picture that may        be followed by one or more RADL pictures but no RASL picture in        decoding order.    -   IDR with no leading picture (IDR_N_LP): NAL unit of an IDR        picture that is not followed by leading picture in decoding        order.    -   CRA: NAL unit of a Clean Random Access (CRA) picture that may be        followed by leading pictures (i.e., either RASL pictures or RADL        pictures or both).    -   RADL: NAL unit of a RADL picture.    -   RASL: NAL unit of a RASL picture.

For VVC, the NAL unit type for TRAP pictures 402 and leading pictures404 are as follows:

-   -   IDR with RADL (IDR_W_RADL): NAL unit of an IDR picture that may        be followed by one or more RADL pictures but no RASL picture in        decoding order.    -   IDR with no leading picture (IDR_N_LP): NAL unit of an IDR        picture that is not followed by leading picture in decoding        order.    -   CRA: NAL unit of a Clean Random Access (CRA) picture that may be        followed by leading pictures (i.e., either RASL pictures or RADL        pictures or both).    -   RADL: NAL unit of a RADL picture.    -   RASL: NAL unit of a RASL picture.

Progressive Intra Refresh/Gradual Decoding Refresh are discussed below.

For low delay applications, it is desirable to avoid coding a picture asan TRAP picture (e.g., TRAP picture 402) due to its relatively largebitrate requirement compared to non-TRAP (i.e., P-/B-) pictures, whichconsequently cause more latency/delays. However, totally avoiding theuse of IRAP may not be possible in all low delay applications. Forexample, for conversational applications such as multi-partyteleconferencing it is necessary to provide regular points in which anew user can join the teleconferencing.

To provide access to a bitstream that allows a new user to join amulti-party teleconferencing application, one possible strategy is touse a progressive intra refresh technique (PIR) instead of using TRAPpictures to avoid having a peak in bitrate. PIR may also be referred toas gradual decoding refresh (GDR). The term PIR and GDR may be usedinterchangeably in the present disclosure.

FIG. 5 illustrates a gradual decoding refresh (GDR) technique 500. Asshown, the GDR technique 500 is depicted using a GDR picture 502, one ormore trailing pictures 504, and a recovery point picture 506 within acoded video sequence 508 of a bitstream. In an embodiment, the GDRpicture 502, the trailing pictures 504, and the recovery point picture506 may define a GDR period in the CVS 508. The CVS 508 is a series ofpictures (or portions thereof) starting with the GDR picture 502 andincludes all pictures (or portions thereof) up to, but not including,the next GDR picture or until the end of the bitstream. The GDR periodis a series of pictures starting with the GDR picture 502 and includesall pictures up to and including the recovery point picture 506.

As shown in FIG. 5 , the GDR technique 500 or principle works over aseries of pictures starting with the GDR picture 502 and ending with therecovery point picture 506. The GDR picture 502 contains arefreshed/clean region 510 containing blocks that have all be codedusing intra prediction (i.e., intra-predicted blocks) and anun-refreshed/dirty region 512 containing blocks that have all be codedusing inter prediction (i.e., inter-predicted blocks).

The trailing picture 504 immediately adjacent to the GDR picture 502contains a refreshed/clean region 510 having a first portion 510A codedusing intra prediction and a second portion 510B coded using interprediction. The second portion 510B is coded by referencing therefreshed/clean region 510 of, for example, a preceeding picture withinthe GDR period of the CVS 508. As shown, the refreshed/clean region 510of the trailing pictures 504 expands as the coding process moves orprogresses in a consistent direction (e.g., from left to right), whichcorrespondingly shrinks the un-refreshed/dirty region 512. Eventually,the recovery point picture 506, which contains only the refreshed/cleanregion 510, is obtained from the coding process. Notably, and as will befurther discussed below, the second portion 510B of the refreshed/cleanregion 510, which is coded as inter-predicted blocks, may only refer tothe refreshed region/clean region 510 in the reference picture.

In HEVC, the GDR technique 500 of FIG. 5 was supported non-normativelyusing a recovery point Supplemental Enhancement Information (SEI)message and a region refresh information SEI message. These two SEImessages do not define how GDR is performed. Rather, the two SEImessages simply provide a mechanism to indicate the first and the lastpictures in the GDR period (i.e., provided by the recovery point SEImessage) and the region that is refreshed (i.e., provided by the regionrefresh information SEI message).

In practice, the GDR technique 500 is performed by using two techniquestogether. Those two techniques are constraint intra prediction (CIP) andencoder constraints for motion vectors. CIP can be used for GDRpurposes, particularly to code the region which is coded only asintra-predicted blocks (e.g., the first portion 510A of therefreshed/clean region 510) because CIP allows the region not usingsamples from the un-refreshed region (e.g., the un-refreshed/dirtyregion 512) to be used for reference. However, the use of CIP causessevere coding performance degradation because the constraint to intrablocks must be applied not only for intra blocks in the refreshedregion, but also to all intra blocks in the picture. Encoder constraintsfor motion vectors restrict the encoder from using any samples in thereference pictures that are located outside refreshed region. Such aconstraint causes non-optimal motion search.

FIG. 6 is a schematic diagram illustrating an undesirable motion search600 when using the encoder restriction to support GDR. As shown, themotion search 600 depicts a current picture 602 and a reference picture604. The current picture 602 and the reference picture 604 each includea refreshed region 606 coded with intra prediction, a refreshed region608 coded with inter prediction, and an unrefreshed region 608. Therefreshed region 604, the refreshed region 606, and the unrefreshedregion 608 are similar to the first portion 510A of the refreshed/cleanregion 510, the second portion 510B of the refreshed/clean region 510,and the un-refreshed/dirty region 512 in FIG. 5 .

During motion search process, the encoder is constrained or preventedfrom selecting any motion vector 610 that results in some of the samplesof the reference block 612 being located outside the refreshed region606. This occurs even when the reference block 612 provides the bestrate-distortion cost criteria when predicting the current block 614 inthe current picture 602. Thus, FIG. 6 illustrates the reason fornon-optimality in the motion search 600 when using the encoderrestriction for supporting GDR.

The JVET contributions JVET-K0212 and JVET-L0160 describe theimplementation of GDR based on the use of CIP and the encoderconstraints approach. The implementation can be summarized as follows:intra prediction mode is forced on a coding unit on a column basis,constrained intra prediction is enabled to ensure reconstruction ofintra CU, motion vectors are constrained to point within the refreshedarea while taking into account an additional margin to avoid errorspreading for filters (6 pixels for instance), and removing formerreference pictures when re-looping the intra column.

The JVET contribution JVET-M0529 proposed a method for indicating that apicture is the first and the last in the GDR period normatively. Theproposed idea works as follows.

Define a new NAL unit with a NAL unit type recovery point indication asa non-video coding layer (VCL) NAL unit. The payload of the NAL unitcontains a syntax element to specify information that can be used toderive the POC value of the last picture in the GDR period. The accessunit that contains the non-VCL NAL unit with the type recovery pointindication is called a recovery point begin (RBP) access unit (AU) andthe picture in the RBP access unit is called an RBP picture. Thedecoding process can start from RBP AU. When decoding starts from RBPAU, all pictures in the GDR period, except the last picture, are notoutput.

Some of the problems with the existing GDR design are discussed.

The existing designs/approaches for supporting GDR have at least thefollowing problems.

The method for normatively defining GDR in NET-M0529 has the followingproblems. The proposed method does not describe how GDR is performed.Instead, the proposed method only provides some signaling for indicatingthe first and the last pictures in the GDR period. For indicating thefirst and the last pictures in the GDR period, a new non-VCL NAL unit isneeded. This is a redundancy as the information contained in therecovery point indication (RPI) NAL unit can simply be included in thetile group header of the first picture in the GDR period. Also, theproposed method is not able to describe which region in the pictures inthe GDR period are the refreshed region and un-refreshed region.

The GDR approach described in JVET-K0212 and JVET-L0160 has thefollowing problems. First, the use of CIP. It is necessary to code arefreshed region with intra-prediction with some constraints to preventany samples from the unrefreshed region being used for spatialreference. When CIP is used, the coding is picture based, which meansthat all intra blocks in the picture must also be coded as CIP intrablocks. This consequently causes performance degradation. Further, theuse of an encoder constraint to limit motion search prevents the encoderfrom choosing the best motion vector when samples of the reference blockassociated with the motion vector are not completely within therefreshed region in the reference picture. Also, the refreshed regionthat is coded only with intra-prediction is not CTU size. Instead, therefreshed region can go smaller than CTU size, down to minimum CU size.This makes the implementation unnecessarily complicated as it may needan indication at the block level.

Disclosed herein are techniques for supporting gradual decoding refresh(GDR) in video coding. The disclosed techniques allow progressive intrarefresh to enable random access without having to use an intra randomaccess point (TRAP) picture. The value of a flag is set to prevent a GDRpicture and any trailing pictures between the GDR picture and a recoverypoint picture in output order from being output when the GDR picture isthe initial picture in decoding order or the initial picture followingthe EOS NAL unit in the decoding order. Setting the flag in this mannerprevents potentially dirty data from being output to a display. That is,the value of the flag setting controls whether potentially dirty datafrom the GDR picture is output or whether the video decoder will waitfor full synchronization to begin displaying data. By having the abilityto restrict dirty data from being output, the coder/decoder (a.k.a.,“codec”) in video coding is improved relative to current codecs. As apractical matter, the improved video coding process offers the user abetter user experience when videos are sent, received, and/or viewed.

In order to solve one or more of the problems discussed above, thepresent disclosure discloses the following aspects. Each of the aspectscan be applied individually, and some of them can be applied incombination.

1) A VCL NAL unit with type GDR_NUT is defined.

a. Pictures with the NAL unit type GDR_NUT are referred to as a GDRpicture, that is, the first picture in the GDR period.

b. A GDR picture has a temporalID equal to 0.

c. An access unit that contains a GDR picture is referred to as a GDRaccess unit. As noted above, an access unit is a set of NAL units. EachNAL unit may contain a single picture.

2) A coded video sequence (CVS) may start with a GDR access unit.

3) A GDR access unit is the first access unit in the CVS when one of thefollowing is true:

a. The GDR access unit is the first access unit in the bitstream.

b. The GDR access unit immediately follows an end-of-sequence (EOS)access unit.

c. The GDR access unit immediately follows an end-of-bitstream (EOB)access unit.

d. A decoder flag so called NoIncorrectPicOutputFlag is associated withthe GDR picture and the value of the flag is set to equal to 1 (i.e.,true) by an entity outside the decoder.

4) When a GDR picture is the first access unit in the CVS, the followingapply:

a. All the reference pictures in the DPB are marked as “unused forreference.”

b. The POC MSB of the picture is set to be equal to 0.

c. The GDR picture and all the pictures that follow the GDR picture inoutput order until the last picture in the GDR period, excluding thelast picture in the GDR period, are not to be output (i.e., marked as“not needed for output”).

5) A flag to specify whether GDR is enabled is signaled in a sequencelevel parameter set (e.g., in the SPS).

a. The flag may be designated gdr_enabled_flag.

b. When the flag is equal to 1, the GDR picture may be present in theCVS. Otherwise, when the flag is equal to 0, GDR is not enabled so thatthe GDR picture is not present in the CVS.

6) Information that can be used to derive the POC value of the lastpicture in the GDR period is signaled in tile group header of GDRpicture.

a. The information is signaled as a delta POC between the last picturein the GDR period and the GDR picture. The information can be signaledusing a syntax element designated recovery_point_cnt.

b. The presence of the syntax element recovery_point_cnt in a tile groupheader may be conditioned upon the value of a gdr_enabled_flag and theNAL unit type of a picture, i.e., the flag is present only when thegdr_enabled_flag is equal to 1 and the nal_unit_type of the NAL unitcontaining the tile group is GDR_NUT.

7) A flag to specify whether a tile group is part of a refreshed regionor not is signaled in tile group header.

a. The flag may be designated refreshed region flag.

b. The presence of the flag may be conditioned upon the value of thegdr_enabled_flag and whether the picture containing the tile group iswithin a GDR period. Thus, the flag is present only when all of thefollowing are true:

i. The value of gdr_enabled_flag is equal to 1.

ii. The POC of the current picture is equal to or greater than the POCvalue of the last GDR picture (when the current picture is the GDRpicture, the last GDR picture is the current picture) and is less thanthe POC of the last picture in the GDR period.

c. When the flag is not present in the tile group header, the value ofthe flag is inferred to be equal to 1.

8) All tile groups having a refreshed_region_flag equal to 1 cover aconnected region. Likewise, all tile groups having therefreshed_region_flag equal to 0 cover the connected region as well.

9) A tile group with a refreshed_region_flag can be of type I (i.e.,intra tile group) or B- or P- (i.e., inter tile group).

10) Each picture starting from the GDR picture to the last picture inthe GDR period contains at least one tile group with arefreshed_region_flag equal to 1.

11) A GDR picture contains at least one tile group with arefreshed_region_flag equal to 1 and a tile_group_type equal to I (i.e.,intra tile group).

12) When the gdr_enabled_flag is equal to 1, information of rectangulartile group, i.e., the number of tile groups and their addresses, isallowed to be signaled in either the picture parameter set (PPS) or intile group header. To do this, a flag is signaled in the PPS to specifywhether rectangular tile group information is present in the PPS or not.This flag may be called rect_tile_group_info_in_pps_flag. This flag maybe constrained to be equal to 1 when the gdr_enabled_flag is equal to 1.

a. In one alternative, instead of signaling whether rectangular tilegroup information is present in the PPS or not, a more general flag canbe signaled in the PPS to specify whether tile group information (i.e.,any types of tile group such as rectangular tile group, raster-scan tilegroup) is present in the PPS.

13) When tile group information is not present in the PPS, it may befurther constrained that the signaling of explicit tile group identifier(ID) information is not present. Explicit tile group ID informationincludes: signaled_tile_group_id_flag,signaled_tile_group_id_length_minus1, and tile_group_id[i].

14) A flag is signaled to specify whether a loop filtering operationacross the boundaries between the refreshed region and un-refreshedregion in a picture is allowed.

a. This flag may be signaled in the PPS and calledloop_filter_across_refreshed_region_enabled_flag.

b. The presence of a loop_filter_across_refreshed_region_enabled_flagmay be conditioned upon the value of aloop_filter_across_tile_enabled_flag. When theloop_filter_across_tile_enabled_flag is equal to 0, theloop_filter_across_refreshed_region_enabled_flag may not be present andits value is inferred to be equal to 0.

c. In one alternative, the flag may be signaled in the tile group headerand its presence may be condition upon the value ofrefreshed_region_flag, i.e., the flag is present only when the value ofrefreshed_region_flag is equal to 1.

15) When it is indicated that a tile group is a refreshed region and itis indicated that loop filter across refreshed region is not allowed,the following apply:

a. Deblocking of the edge at the border of the tile group is notperformed when the neighbor tile group that shares the edge is anun-refreshed tile group.

b. Sample adaptive offset (SAO) process for blocks at the boundary ofthe tile group does not use any samples from outside the refreshedregion boundary.

c. Adaptive loop filtering (ALF) process for blocks at the boundary ofthe tile group does not use any samples from outside the refreshedregion boundary.

16) When the gdr_enabled_flag is equal to 1, each picture is associatedwith variables to determine the boundaries of the refreshed region inthe picture. These variables may be called:

a. PicRefreshedLeftBoundaryPos for the left boundary position of therefreshed region in the picture.

b. PicRefreshedRightBoundaryPos for the right boundary position of therefreshed region in the picture.

c. PicRefreshedTopBoundaryPos for the top boundary position of therefreshed region in the picture.

d. PicRefreshedBotBoundaryPos for the bottom boundary position of therefreshed region in the picture.

17) The boundaries of a refreshed region in a picture may be derived.The boundaries of the refreshed region of the picture are updated by thedecoder after a tile group header is parsed and the value ofrefreshed_region_flag of the tile group is equal to 1.

18) In one alternative to solution 17), the boundaries of the refreshedregion in the picture are signaled explicitly in each tile group of thepicture.

a. A flag may be signaled to indicate whether the picture that the tilegroup belongs to contains an un-refreshed region. When it is specifiedthat the picture does not contain the un-refreshed region, no refreshedboundaries information is signaled and can simply be inferred to beequal to the picture boundaries.

19) For the current picture, the boundaries of the refreshed region areused in an in-loop filters process as follows:

a. For the deblocking process, to determine the edge of the refreshedregion for deciding whether the edge needs to be deblocked or not.

b. For the SAO process, to determine the border of the refresh region sothat the clipping process can be applied to avoid using samples from theun-refreshed region when loop filter across the refreshed region is notallowed.

c. For the ALF process, to determine the border of the refresh region sothat the clipping process can be applied to avoid using samples fromun-refreshed region when loop filter across refreshed region is notallowed.

20) For the motion compensation process, the information aboutboundaries of the refreshed region, particularly the boundaries of therefreshed region in the reference picture, is used as follows: when thecurrent block in the current picture is in the tile group with arefreshed_region_flag equal to 1 and the reference block is in areference picture that contains an un-refreshed region, the followingapply:

a. The motion vector from the current block to that reference picture isclipped by the boundaries of the refreshed region in that referencepicture.

b. For a fractional interpolation filter for samples in that referencepicture, it is clipped by the boundaries of the refreshed region in thatreference picture.

A detailed description of embodiments of the present disclosure isprovided. The description is relative to the basis text, which is theJVET contribution JVET-M1001-v5. That is, only the delta is described,while the texts in the basis text that are not mentioned below apply asthey are. Modified text relative to the basis text is italicized.

Definitions are provided.

3.1 clean random access (CRA) picture: An IRAP picture for which eachVCL NAL unit has nal_unit_type equal to CRA_NUT.

NOTE—A CRA picture does not refer to any pictures other than itself forinter prediction in its decoding process, and may be the first picturein the bitstream in decoding order, or may appear later in thebitstream. A CRA picture may have associated RADL or RASL pictures. Whena CRA picture has a NoIncorrectPicOutputFlag equal to 1, the associatedRASL pictures are not output by the decoder, because the RASL picturesmay not be decodable, as they may contain references to pictures thatare not present in the bitstream.

3.2 coded video sequence (CVS): A sequence of access units thatcomprises, in decoding order, of an IRAP access unit withNoIncorrectPicOutputFlag equal to 1 or a GDR access unit withNoIncorrectPicOutputFlag equal to 1, followed by zero or more accessunits that are not TRAP access units with NoIncorrectPicOutputFlag equalto 1 or GDR access unit with NoIncorrectPicOutputFlag equal to 1,including all subsequent access units up to but not including anysubsequent access unit that is an IRAP access unit withNoIncorrectPicOutputFlag equal to 1 or GDR access unit withNoIncorrectPicOutputFlag equal to 1.

NOTE 1—An TRAP access unit may be an IDR access unit or a CRA accessunit. The value of NoIncorrectPicOutputFlag is equal to 1 for each IDRaccess unit and each CRA access unit that is the first access unit inthe bitstream in decoding order, is the first access unit that followsan end of sequence NAL unit in decoding order, or hasHandleCraAsCvsStartFlag equal to 1.

NOTE 2—The value of NoIncorrectPicOutputFlag is equal to 1 for each GDRaccess unit that is the first access unit in the bitstream in decodingorder, is the first access unit that follows an end of sequence NAL unitin decoding order, or has HandleGdrAsCvsStartFlag equal to 1.

3.3 gradual decoding refresh (GDR) access unit: An access unit in whichthe coded picture is a GDR picture.

3.4 gradual decoding refresh (GDR) picture: A picture for which each VCLNAL unit has nal_unit_type equal to GDR_NUT

3.5 random access skipped leading (RASL) picture: A coded picture forwhich each VCL NAL unit has nal_unit_type equal to RASL_NUT.

NOTE—All RASL pictures are leading pictures of an associated CRApicture. When the associated CRA picture has NoIncorrectPicOutputFlagequal to 1, the RASL picture is not output and may not be correctlydecodable, as the RASL picture may contain references to pictures thatare not present in the bitstream. RASL pictures are not used asreference pictures for the decoding process of non-RASL pictures. Whenpresent, all RASL pictures precede, in decoding order, all trailingpictures of the same associated CRA picture.

Sequence parameter set raw byte sequence payload (RBSP) syntax andsemantics.

seq_parameter_set_rbsp( ) { Descriptor  ...   

u(1)  ...

gdr_enabled_flag equal to 1 specifies GDR pictures may be present in thecoded video sequence. gdr_enabled_flag equal to 0 specifies that GDRpictures are not present in the coded video sequence.

Picture parameter set RB SP syntax and semantics.

pic_parameter_set_rbsp( ) { Descriptor  ...  single_tile_in_pic_flagu(1)  if( !single_tile_in_pic_flag ) {   ... ue(v)  single_tile_per_tile_group_flag u(1)   if(!single_tile_per_tile_group_flag )    rect_tile_group_flag u(1)   if(rect_tile_group_flag )     

 

u(1)   if( rect_tile_group_flag && !single_tile_per_tile_group_flag &&      rect_tile_group_info_in_pps_flag ) {   num_tile_groups_in_pic_minus1 ue(v)    for( i = 0; i <=num_tile_groups_in_pic_minus1; i++ ) {     if( i > 0 )     top_left_tile_idx[ i ] u(v)     bottom_right_tile_idx[ i ] u(v)   }   }   loop_filter_across_tiles_enabled_flag u(1)   if(loop_filter_across_tiles_enabled_flag )   loop_filter_across_tile_groups_enabled_flag u(1)   if(loop_filter_across_tile_groups_enabled_flag )     

 

u(1)  }  ...

rect_tile_group_info_in_pps_flag equal to 1 specifies that therectangular tile group information is signalled in the PPS.rect_tile_group_info_in_pps_flag equal to 0 specifies that therectangular tile group information is not signalled in the PPS.

It is a requirement of bitstream conformance that the value ofrect_tile_group_info_in_pps_flag shall be equal to 0 when the value ofgdr_enabled_flag in the active SPS is equal to 0.

loop_filter_across_refreshed_region_enabled_flag equal to 1 specifiesthat in-loop filtering operations may be performed across boundaries oftile group with refreshed_region_flag equal to 1 in pictures referringto the PPS. loop_filter_across_refreshed_region_enabled_flag equal to 0specifies that in-loop filtering operations are not performed acrossboundaries of tile group with refreshed_region_flag equal to 1 inpictures referring to the PPS. The in-loop filtering operations includethe deblocking filter, sample adaptive offset filter, and adaptive loopfilter operations. When not present, the value ofloop_filter_across_refreshed_region_enabled_flag is inferred to be equalto 0.

signalled_tile_group_id flag equal to 1 specifies that the tile group IDfor each tile group is signalled. signalled_tile_group_index_flag equalto 0 specifies that tile group IDs are not signalled. When not present,the value of signalled_tile_group_index_flag is inferred to be equal to0.

signalled_tile_group_id_length_minus1 plus 1 specifies the number ofbits used to represent the syntax element tile_group_id[i] when present,and the syntax element tile_group_address in tile group headers. Thevalue of signalled_tile_group_index_length_minus1 shall be in the rangeof 0 to 15, inclusive. When not present, the value ofsignalled_tile_group_index_length_minus1 is inferred as follows:

If rect_tile_group_info_in_pps_flag is equal to 1, Ceil(Log2(num_tile_groups_in_pic_minus1+1))−1.

Otherwise, Cell(Log 2(NumTilesInPic))−1

General tile group header syntax and semantics.

tile_group_header( ) { Descriptor  ...  if( rect_tile_group_flag ∥NumTilesInPic > 1 )   tile_group_address u(v)  if(!rect_tile_group_info_in_pps_flag && !single_tile_per_tile_group_flag ){    if( rect_tile_group_flag )     

u(v)   else    num_tiles_in_tile_group_minus1 ue(v)  }  tile_group_typeue(v)  tile_group_pic_order_cnt_lsb u(v)   if( gdr_enabled_flag ) {   if( nal_unit_type = = GDR_NUT )     

se (v)    if( PicOrderCntVal >= LastGDRPocVal &&     PicOrderCntVal <RecoveryPointPocVal )     

u(1)   

 ... )

tile_group_address specifies the tile address of the first tile in thetile group. When not present, the value of tile_group_address isinferred to be equal to 0.

If rect_tile_group_flag is equal to 0, the following applies:

-   -   The tile_group_address is the tile ID as specified by Equation        6-7.    -   The length of tile_group_address is Ceil(Log 2 (NumTilesInPic))        bits.    -   The value of tile_group_address shall be in the range of 0 to        NumTilesInPic−1, inclusive.

Else if rect_tile_group_flag is equal to 1 andrect_tile_group_info_in_pps is equal to 0, the following applies:

-   -   The tile_group_address is the tile index of the tile located at        the top-left corner of the i-th tile group.    -   The length of tile_group_address is        signalled_tile_group_index_length_minus1+1 bits.    -   If signalled_tile_group_id_flag is equal to 0, the value of        tile_group_address shall be in the range of 0 to        NumTilesInPic−1, inclusive. Otherwise, the value of        tile_group_address shall be in the range of 0 to to        2^((signalled_tile_group_index_length_minus1+1))−1, inclusive.

Else (rect_tile_group_flag is equal to 1 and rect_tile_group_info_in_ppsis equal to 1), the following applies:

-   -   The tile_group_address is the tile group ID of the tile group.    -   The length of tile_group_address is        signalled_tile_group_index_length_minus1+1 bits.    -   If signalled_tile_group_id_flag is equal to 0, the value of        tile_group_address shall be in the range of 0 to        num_tile_groups_in_pic_minus1, inclusive. Otherwise, the value        of tile_group_address shall be in the range of 0 to        2^((signalled_tile_group_index_length_minus1+1))−1, inclusive.

bottom_right_tile_id specifies the tile index of the tile located at thebottom-right corner of the tile group. Whensingle_tile_per_tile_group_flag is equal to 1 bottom_right_tile_id isinferred to be equal to tile_group_address. The length of thebottom_right_tile_id syntax element is Cell(Log 2(NumTilesInPic)) bits.

The variable NumTilesInCurrTileGroup, which specifies the number oftiles in the current tile group, TopLeftTileIdx which specifies the tileindex of the top left tile of the tile group, BottomRightTileIdx whichspecifies the tile index of the bottom right tile of the tile group, andTgTileIdx[i], which specifies the tile index of the i-th tile in thecurrent tile group, are derived as follows:

 if( rect_tile_group_flag ) {   if ( tile_group_info_in_pps ) {   tileGroupIdx = 0    while( tile_group_address != rect_tile_group_id[tileGroupIdx ] )     tileGroupIdx++    tileIdx = top_left_tile_idx[tileGroupIdx ]    BottomRightTileIdx = bottom_right_tile_idx[tileGroupIdx ]   } else {    tileIdx = tile_group_address   BottomRightTileIdx = bottom_right_tile_id   }   TopLeftTileIdx =tileIdx   deltaTileIdx = BottomRightTileIdx − TopLeftTileIdx  NumTileRowsInTileGroupMinus1 = deltaTileIdx / (num_tile_columns_minus1 + 1 ) (7-35)   NumTileColumnsInTileGroupMinus1 =deltaTileIdx % ( num_tile_columns_minus1 + 1 )   NumTilesInCurrTileGroup= ( NumTileRowsInTileGroupMinus1 + 1 ) *       (NumTileColumnsInTileGroupMinus1 + 1 )   for( j = 0, tIdx = 0; j <NumTileRowsInTileGroupMinus1 + 1;      j++, tileIdx +=num_tile_columns_minus1 + 1 )   for( i = 0, currTileIdx = tileIdx; i <NumTileColumnsInTileGroupMinus1 + 1;        i++, currTileIdx++, tIdx++ )    TgTileIdx[ tIdx ] = currTileIdx  } else {   NumTilesInCurrTileGroup= num_tiles_in_tile_group_minus1 + 1   TgTileIdx[ 0 ] =tile_group_address   for( i = 1; i < NumTilesInCurrTileGroup; i++ )   TgTileIdx[ i ] = TgTileIdx[ i − 1 ] + 1  }

specifies the recovery point of decoded pictures in output order. Ifthere is a picture picA that follows the current picture (i.e., a GDRpicture) in decoding order in the CVS and that has PicOrderCntVal equalto the PicOrderCntVal of the current picture plus the value ofrecovery_poc_cnt, the picture picA is referred to as the recovery pointpicture. Otherwise, the first picture in output order that hasPicOrderCntVal greater than the PicOrderCntVal of the current pictureplus the value of recovery_poc_cnt is referred to as the recovery pointpicture. The recovery point picture shall not precede the currentpicture in decoding order. All decoded pictures in output order areindicated to be correct or approximately correct in content starting atthe output order position of the recovery point picture. The value ofrecovery_poc_cnt shall be in the range of −MaxPicOrderCntLsb/2 toMaxPicOrderCntLsb/2−1, inclusive.

The value RecoveryPointPocVal is derived as follows:

RecoveryPointPocVal=PicOrderCntVal+recovery_poc_cnt

equal to 1 specifies that the decoding of the tile group producescorrect reconstucted samples values regardless whether the value ofNoIncorrectPicOutputFlag of the associated GDR. refreshed_region_flagequal to 0 specifies that the decoding of the tile group may produceincorrect reconstucted samples values when it starts from the associatedGDR with NoIncorrectPicOutputFlag equal to 1. When not present, thevalue of refreshed_region_flag is inferred to be equal to 1.

NOTE x—The current picture itself can be the GDR picture withNoIncorrectPicOutputFlag equal to 1.

  The tile group refreshed boundaries are derived as follows: tileColIdx = TopLeftTileIdx % ( num_tile_columns_minus1 + 1 ) tileRowIdx = TopLeftTileIdx / ( num_tile_columns_minus1 + 1 ) TGRefreshedLeftBoundary = ColBd[ tileColIdx ] << CtbLog2SizeY TGRefreshedTopBoundary = RowBd[ tileRowIdx ] << CtbLog2SizeY tileColIdx = BottomRightTileIdx % ( num_tile_columns_minus1 + 1 ) tileRowIdx = BottomRightTileIdx / ( num_tile_columns_minus1 + 1 ) TGRefreshedRightBoundary = ( ( ColBd[ tileColIdx ] + ColWidth[tileColIdx ]) << CtbLog2SizeY) − 1  TGRefreshedRightBoundary =TGRefreshedRightBoundary > pic_width_in_luma_samples ?        pic_width_in_luma_samples   : TGRefreshedRightBoundary TGRefreshedBotBoundary = ( ( RowBd[ tileRowIdx ] + RowHeight[tileRowIdx ] ) << CtbLog2SizeY ) − 1  TGRefreshedBotBoundary =TGRefreshedBotBoundary > pic_height_in_luma_samples ?        pic_height_in_luma_samples   : TGRefreshedBotBoundary

NAL Unit Header Semantics.

TABLE 7-1 NAL unit type codes and NAL unit type classes Name of NAL unitnal_unit_type nal_unit_type Content of NAL unit and RBSP syntaxstructure type class  0 TRAIL_NUT Coded tile group of a non- STSAtrailing picture VCL tile_group_layer_rbsp( )  1 STSA_NUT Coded tilegroup of an STSA picture VCL tile_group_layer_rbsp( )  2 RASL_NUT Codedtile group of a RASL picture VCL tile_group_layer_rbsp( )  3 RADL_NUTCoded tile group of a RADL picture VCL tile_group_layer_rbsp( )  4 . . .7 RSV_VCL_4 . . . Reserved non-IRAP VCL NAL unit types VCL RSV_VCL_7  8IDR_W_RADL Coded tile group of an IDR picture VCL  9 IDR_N_LPtile_group_layer_rbsp( ) 10 CRA_NUT Coded tile group of a CRA pictureVCL tile_group_layer_rbsp( ) 11 RSV_IRAP_VCL11 Reserved IRAP VCL NALunit types VCL 12 RSV_IRAP_VCL12 13 GDR_NUT Coded tile group of a GDRpicture VCL tile_group_layer_rbsp( ) 14 . . . 15 RSV_VCL14 . . .Reserved non-IRAP VCL NAL unit types VCL RSV_VCL15 16 SPS_NUT Sequenceparameter set non-VCL seq_parameter_set_rbsp( ) 17 PPS_NUT Pictureparameter set non-VCL pic_parameter_set_rbsp( ) 18 APS_NUT Adaptationparameter set non-VCL adaptation_parameter_set_rbsp( ) 19 AUD_NUT Accessunit delimiter non-VCL access_unit_delimiter_rbsp( ) 20 EOS_NUT End ofsequence non-VCL end_of_seq_rbsp( ) 21 EOB_NUT End of bitstream non-VCLend_of_bitstream_rbsp( ) 22, 23 PREFIX_SEI_NUT Supplemental enhancementinformation non-VCL SUFFIX_SEI_NUT sei_rbsp( ) 24 . . . 27 RSV_NVCL24 .. . Reserved non-VCL NAL unit types non-VCL RSV_NVCL27 28 . . . 31UNSPEC28 . . . Unspecified non-VCL NAL unit types non-VCL UNSPEC31

. . .

When nal_unit_type is equal to GDR_NUT, the coded tile group belongs toa GDR picture, TemporalId shall be equal to 0.

Order of access units and association to CVSs is discussed.

A bitstream conforming to this Specification (i.e., NET contributionJVET-M1001-v5) includes one or more CVSs.

A CVS includes one or more access units. The order of NAL units andcoded pictures and their association to access units is described inclause 7.4.2.4.4.

The first access unit of a CVS is one of the following:

-   -   an TRAP access unit with NoBrokenPictureOutputFlag equal to 1.    -   a GDR access unit with NoIncorrectPicOuqmtFlag equal to 1.

It is a requirement of bitstream conformance that, when present, thenext access unit after an access unit that contains an end of sequenceNAL unit or an end of bitstream NAL unit shall be one of the following:

-   -   an IRAP access unit, which may be an IDR access unit or a CRA        access unit.    -   a GDR access unit.

8.1.1 Decoding process for a coded picture is discussed.

. . .

When the current picture is an TRAP picture, the following applies:

-   -   When the current picture is an IDR picture, the first picture in        the bitstream in decoding order, or the first picture that        follows an end of sequence NAL unit in decoding order, the        variable NoIncorrectPicOutputFlag is set equal to 1.    -   Otherwise, when some external means not specified in this        Specification (e.g., a user input) is available to set the        variable HandleCraAsCvsStartFlag to a value for the current        picture, the variable HandleCraAsCvsStartFlag is set equal to        the value provided by the external means and the variable        NoIncorrectPicOutputFlag is set equal to        HandleCraAsCvsStartFlag.    -   Otherwise, the variable HandleCraAsCvsStartFlag is set equal to        0 and the variable NoIncorrectPicOutputFlag is set equal to 0.

When the current picture is a GDR picture, the following applies:

-   -   When the current picture is a GDR picture, the first picture in        the bitstream in decoding order, or the first picture that        follows an end of sequence NAL unit in decoding order, the        variable NoIncorrectPicOutputFlag is set equal to 1.    -   Otherwise, when some external means not specified in this        Specification is available to set the variable        HandleGdrAsCvsStartFlag to a value for the current picture, the        variable HandleGdrAsCvsStartFlag is set equal to the value        provided by the external means and the variable        NoIncorrectPicOutputFlag is set equal to        HandleGdrAsCvsStartFlag.    -   Otherwise, the variable HandleGdrAsCvsStartFlag is set equal to        0 and the variable NoIncorrectPicOutputFlag is set equal to 0.

. . .

The decoding process operates as follows for the current pictureCurrPic:

1. The decoding of NAL units is specified in clause 8.2.

2. The processes in clause 8.3 specify the following decoding processesusing syntax elements in the tile group header layer and above:

-   -   Variables and functions relating to picture order count are        derived as specified in clause 8.3.1. This needs to be invoked        only for the first tile group of a picture.    -   At the beginning of the decoding process for each tile group of        a non-IDR picture, the decoding process for reference picture        lists construction specified in clause 8.3.2 is invoked for        derivation of reference picture list 0 (RefPicList[0]) and        reference picture list 1 (RefPicList[1]).    -   The decoding process for reference picture marking in clause        8.3.3 is invoked, wherein reference pictures may be marked as        “unused for reference” or “used for long-term reference.” This        is invoked only for the first tile group of a picture.    -   PicOutputFlag is set as follows:    -   When one of the following conditions is true, PictureOutputFlag        is set equal to 0:    -   the current picture is a RASL picture and        NoIncorrectPicOutputFlag of the associated IRAP picture is equal        to 1.    -   gdr_enabled_flag is equal to 1 and the current picture is GDR        picture with NoIncorrectPicOutputFlag equal to 1.    -   gdr_enabled_flag is equal to 1 and the current picture contains        one or more tile group with refreshed region_flag equal to 0 and        NoBrokenPictureOutputFlag of the associated GDR picture is equal        to 1.    -   Otherwise, PicOutputFlag is set equal to 1.

3. The decoding processes is invoked for coding tree units, scaling,transform, in-loop filtering, etc.

4. After all tile groups of the current picture have been decoded, thecurrent decoded picture is marked as “used for short-term reference.”

Decoding process for picture order count is discussed.

Output of this process is PicOrderCntVal, the picture order count of thecurrent picture.

Each coded picture is associated with a picture order count variable,denoted as PicOrderCntVal.

When the current picture is not an TRAP picture withNoIncorrectPicOutputFlag equal to 1 or a GDR picture withNoIncorrectPicOutputFlag equal to 1, the variables prevPicOrderCntLsband prevPicOrderCntMsb are derived as follows:

-   -   Let prevTid0Pic be the previous picture in decoding order that        has TemporalId equal to 0 and that is not a RASL or RADL        picture.    -   The variable prevPicOrderCntLsb is set equal to        tile_group_pic_order_cnt_lsb of prevTid0Pic.    -   The variable prevPicOrderCntMsb is set equal to PicOrderCntMsb        of prevTid0Pic.    -   The variable PicOrderCntMsb of the current picture is derived as        follows:    -   When the current picture is an IRAP picture with        NoIncorrectPicOutputFlag equal to 1 or a GDR picture with        NoIncorrectPicOutputFlag equal to 1, PicOrderCntMsb is set equal        to 0.    -   Otherwise, PicOrderCntMsb is derived as follows:    -   if((tile_group_pic_order_cnt_lsb<prevPicOrderCntLsb) &&        ((prevPicOrderCntLsb−tile_group_pic_order_cnt_lsb)>=(MaxPicOrderCntLsb/2)))        PicOrderCntMsb=prevPicOrderCntMsb+MaxPicOrderCntLsb  (8-1)    -   else if((tile_group_pic_order_cnt_lsb>prevPicOrderCntLsb) &&        ((tile_group_pic_order_cnt_lsb−prevPicOrderCntLsb)>(MaxPicOrderCntLsb/2)))        PicOrderCntMsb=prevPicOrderCntMsb−MaxPicOrderCntLsb    -   else    -   PicOrderCntMsb=prevPicOrderCntMsb    -   PicOrderCntVal is derived as follows:        PicOrderCntVal=PicOrderCntMsb+tile_group_pic_order_cnt_lsb  (8-2)

NOTE 1—All IRAP pictures with NoIncorrectPicOutputFlag equal to 1 willhave PicOrderCntVal equal to tile_group_pic_order_cnt_lsb since for TRAPpictures with NoIncorrectPicOutputFlag equal to 1 PicOrderCntMsb is setequal to 0.

NOTE 1—All GDR pictures with NoIncorrectPicOutputFlag equal to 1 willhave PicOrderCntVal equal to tile_group_pic_order_cnt_lsb since for GDRpictures with NoIncorrectPicOutputFlag equal to 1 PicOrderCntMsb is setequal to 0.

The value of PicOrderCntVal shall be in the range of −2³¹ to 2³¹−1,inclusive.

When the current picture is a GDR picture, the value of LastGDRPocVal isset to be equal to PicOrderCntVal.

Decoding process for picture refreshed boundary position is discussed.

This process is invoked only when gdr_enabled_flag is equal to 1.

This process is invoked after the tile group header parsing iscompleted.

Output of this process is PicRefreshedLeftBoundaryPos,PicRefreshedRightBoundaryPos, PicRefreshedTopBoundaryPos, andPicRefreshedBotBoundaryPos, the boundary position of refreshed region ofthe current picture.

Each coded picture is associated with a set of refreshed region boundarypositions variables, denoted as PicOrderCntVal.

PicRefreshedLeftBoundaryPos, PicRefreshedRightBoundaryPos,PicRefreshedTopBoundaryPos, and PicRefreshedBotBoundaryPos are derivedas follows:

If the tile group is the first received tile group of the currentpicture with refreshed_region_flag equal to 1, the following apply:

PicRefreshedLeftBoundaryPos=TGRefreshedLeftBoundary

PicRefreshedRightBoundaryPos=TGRefreshedRightBoundary

PicRefreshedTopBoundaryPos=TGRefreshedTopBoundary

PicRefreshedBotBoundaryPos=TileGroupBotBoundary

-   -   Else if refreshed_region_flag is equal to 1, the followings        apply:

PicRefreshedLeftBoundaryPos=TGRefreshedLeftBoundary<PicRefreshedLeftBoundaryPos?

-   -   TGRefreshedLeftBoundary:PicRefreshedLeftBoundaryPos

PicRefreshedRightBoundaryPos=TGRefreshedRightBoundary>PicRefreshedRightBoundaryPos?

-   -   TGRefreshedRightBoundary:PicRefreshedRightBoundaryPos

PicRefreshedTopBoundaryPos=TGRefreshedTopBoundary<PicRefreshedTopBoundaryPos?

-   -   TGRefreshedTopBoundary:RefreshedRegionTopBoundaryPos

PicRefreshedBotBoundaryPos=TileGroupBotBoundary>PicRefreshedBotBoundaryPos?

-   -   TileGroupBotBoundary:PicRefreshedBotBoundaryPos

Decoding process for reference picture lists construction is discussed.

. . .

It is a requirement of bitstream conformance that for each currentpicture that is not an TRAP picture with NoIncorrectPicOutputFlag equalto 1 or a GDR picture with NoIncorrectPicOutputFlag equal to 1, thevalue of maxPicOrderCnt−minPicOrderCnt shall be less thanMaxPicOrderCntLsb/2.

. . .

Decoding Process for Reference Picture Marking

. . .

If the current picture is an TRAP picture with NoIncorrectPicOutputFlagequal to 1 or a GDR picture with NoIncorrectPicOutputFlag equal to 1,all reference pictures currently in the DPB (if any) are marked as“unused for reference.”

. . .

Derivation process for temporal luma motion vector prediction isdiscussed.

. . .

The variable currCb specifies the current luma coding block at lumalocation (xCb, yCb).

The variables mvLXCol and availableFlagLXCol are derived as follows:

-   -   If tile_group_temporal_mvp_enabled_flag is equal to 0, both        components of mvLXCol are set equal to 0 and availableFlagLXCol        is set equal to 0.    -   Otherwise (tile_group_temporal_mvp_enabled_flag is equal to 1),        the following ordered steps apply:

1. The bottom right collocated motion vector is derived as follows:xColBr=xCb+cbWidth  (8-414)yColBr=yCb+cbHeight  (8-415)

leftBoundaryPos=gdr_enabled_flag?

-   -   PicRefreshedLeftBoundaryPos of picture referred to by        RefPicList[X][refIdxLX]        0  (8-415)

topBoundaryPos=gdr_enabled_flag?

-   -   PicRefreshedTopBoundaryPos of picture referred to by        RefPicList[X][refIdxLX]        0  (8-415)

rightBoundaryPos=gdr_enabled_flag?

-   -   PicRefreshedRightBoundaryPos of picture referred to by        RefPicList[X][refIdxLX]:        pic_width_in_luma_samples  (8-415)

botBoundaryPos=gdr_enabled_flag?

-   -   PicRefreshedBotBoundaryPos of picture referred to by        RefPicList[X][refIdxLX]:        pic_height_in_luma_samples  (8-415)    -   If yCb>>CtbLog 2SizeY is equal to yColBr>>CtbLog 2SizeY, yColBr        is in the range from topBoundaryPos to botBoundaryPos,        inclusive, and xColBr is in the range from leftBoundaryPos to        rightBoundaryPos, inclusive, the following applies:    -   The variable colCb specifies the luma coding block covering the        modified location given by ((xColBr>>3)<<3, (yColBr>>3)<<3)        inside the collocated picture specified by ColPic.    -   The luma location (xColCb, yColCb) is set equal to the top-left        sample of the collocated luma coding block specified by colCb        relative to the top-left luma sample of the collocated picture        specified by ColPic.    -   The derivation process for collocated motion vectors as        specified in clause 8.5.2.12 is invoked with currCb, colCb,        (xColCb, yColCb), refIdxLX and sbFlag set equal to 0 as inputs,        and the output is assigned to mvLXCol and availableFlagLXCol.    -   Otherwise, both components of mvLXCol are set equal to 0 and        availableFlagLXCol is set equal to 0.

2. . . .

Luma sample bilinear interpolation process is discussed.

Inputs to this process are:

-   -   a luma location in full-sample units (xInt_(L), yInt_(L)),    -   a luma location in fractional-sample units (xFrac_(L),        yFrac_(L)),    -   the luma reference sample array refPicLX_(L).    -   the refreshed region boundaries of the reference picture        PicRefreshedLeftBoundaryPos, PicRefreshedTopBoundaryPos,        PicRefreshedRightBoundaryPos, and PicRefreshedBotBoundaryPos.

. . .

The luma locations in full-sample units (xInt_(i), yInt_(i)) are derivedas follows for i=0 . . . 1:

-   -   If gdr_enabled_flag is equal to 1, the following applies:        xInt_(i)=Clip3(PicRefreshedLeftBoundaryPos,PicRefreshedRightBoundaryPos,xInt_(L)        +i)   (8-458)        yInt_(i)=Clip3(PicRefreshedTopBoundaryPos,PicRefreshedBotBoundaryPos,yInt_(L)        +i)   (8-458)    -   Otherwise (gdr_enabled_flag is equal to 0), the following        applies:

xInt_(i)=sps_ref_wraparound_enabled_flag?ClipH((sps_ref_wraparound_offset_minus1+1)*MinCbSizeY,picW,(xInt_(L)+i))   (8-459)

Clip3(0, picW−1, xInt_(L)+i)yInt_(i)=Clip3(0,picH−1,yInt_(L) +i)  (8-460)

. . .

Luma sample 8-tap interpolation filtering process is discussed.

Inputs to this process are:

-   -   a luma location in full-sample units (xInt_(L), yInt_(L)),    -   a luma location in fractional-sample units (xFrac_(L),        yFrac_(L)),    -   the luma reference sample array refPicLX_(L),    -   a list padVal[dir] with dir=0,1 specifying reference sample        padding direction and amount.    -   the refreshed region boundaries of the reference picture        PicRefreshedLeftBoundaryPos, PicRefreshedTopBoundaryPos,        PicRefreshedRightBoundaryPos, and PicRefreshedBotBoundaryPos.

. . .

The luma locations in full-sample units (xInt_(i), yInt_(i)) are derivedas follows for i=0 . . . 7:

-   -   If gdr_enabled_flag is equal to 1, the following applies:        xInt_(i)=Clip3(PicRefreshedLeftBoundaryPos,PicRefreshedRightBoundaryPos,xInt_(L)        +i−3)  (8-830)        yInt_(i)=Clip3(PicRefreshedTopBoundaryPos,PicRefreshedBotBoundaryPos,yInt_(L)        +i−3)  (8-830)    -   Otherwise (gdr_enabled_flag is equal to 0), the following        applies:

xInt_(i)=sps_ref_wraparound_enabled_flag?ClipH((sps_ref_wraparound_offset_minus1+1)*MinCbSizeY,picW,xInt_(L)+i−3):  (8-831)

Clip3(0, picW−1, xInt_(L)+i−3)yInt_(i)=Clip3(0,picH−1,yInt_(L) +i−3)  (8-832)

Chroma sample interpolation process is discussed.

Inputs to this process are:

-   -   a chroma location in full-sample units (xInt_(C), yInt_(C)),    -   a chroma location in 1/32 fractional-sample units (xFrac_(C),        yFrac_(C)),    -   the chroma reference sample array refPicLX_(C).    -   the refreshed region boundaries of the reference picture        PicRefreshedLeftBoundaryPos, PicRefreshedTopBoundaryPos,        PicRefreshedRightBoundaryPos, and PicRefreshedBotBoundaryPos.

. . .

The variable xOffset is set equal to(sps_ref_wraparound_offset_minus1+1)*MinCbSizeY)/SubWidthC.

The chroma locations in full-sample units (xInt_(i), yInt_(i)) arederived as follows for i=0 . . . 3:

-   -   If gdr_enabled_flag is equal to 1, the following applies:        xInt_(i)=Clip3(PicRefreshedLeftBoundaryPos/SubWidthC,PicRefreshedRightBoundaryPos/SubWidthC,xInt_(L)        +i)  (8-844)        yInti=Clip3(PicRefreshedTopBoundaryPos/SubHeightC,PicRefreshedBotBoundaryPos/SubHeightC,yInt_(L)        +i)  (8-844)    -   Otherwise (gdr_enabled_flag is equal to 0), the following        applies:        xInt_(i)=sps_ref_wraparound_enabled_flag?ClipH(xOffset,picW _(C)        ,xInt_(C) +i−1):   (8-845)

Clip3(0, picW_(C)−1, xInt_(C)+i−1)yInt_(i)=Clip3(0,picH _(C)−1,yInt_(C) +i−1)  (8-846)

Deblocking filter process is discussed.

General Process.

. . .

The deblocking filter process is applied to all coding subblock edgesand transform block edges of a picture, except the following types ofedges:

-   -   Edges that are at the boundary of the picture,    -   Edges that coincide with upper boundary of tile group tgA when        all the following are satisfied:    -   gdr_enabled_flag is equal to 1    -   loop_filter_across_refreshed_region_enabled_flag equal to 0    -   the edges coincide with bottom boundary of tile group tgB and        the value of refreshed_region_flag of tgB is different from the        value of refreshed_region_flag of tgA    -   Edges that coincide with left boundary of tile group tgA when        all the following are satisfied:    -   gdr_enabled_flag is equal to 1    -   loop_filter_across_refreshed_region_enabled_flag equal to 0    -   the edges coincide with right boundary of tile group tgB and the        value of refreshed_region_flag of tgB is different from the        value of refreshed_region_flag of tgA    -   Edges that coincide with tile boundaries when        loop_filter_across_tiles_enabled_flag is equal to 0,    -   Edges that coincide with upper or left boundaries of tile groups        with tile_group_loop_filter_across_tile_groups_enabled_flag        equal to 0 or tile_group_deblocking_filter_disabled_flag equal        to 1,    -   Edges within tile groups with        tile_group_deblocking_filter_disabled_flag equal to 1,    -   Edges that do not correspond to 8×8 sample grid boundaries of        the considered component,    -   Edges within chroma components for which both sides of the edge        use inter prediction,    -   Edges of chroma transform blocks that are not edges of the        associated transform unit.    -   Edges across the luma transform blocks of a coding unit that has        an IntraSubPartitionsSplit value not equal to ISP_NO_SPLIT.

Deblocking filter process for one direction is discussed.

. . .

For each coding unit with coding block width log 2CbW, coding blockheight log 2CbH and location of top-left sample of the coding block(xCb, yCb), when edgeType is equal to EDGE_VER and xCb % 8 is equal 0 orwhen edgeType is equal to EDGE_HOR and yCb % 8 is equal to 0, the edgesare filtered by the following ordered steps:

1. The coding block width nCbW is set equal to 1<<log 2CbW and thecoding block height nCbH is set equal to 1<<log 2CbH

2. The variable filterEdgeFlag is derived as follows:

-   -   If edgeType is equal to EDGE_VER and one or more of the        following conditions are true, filterEdgeFlag is set equal to 0:    -   The left boundary of the current coding block is the left        boundary of the picture.    -   The left boundary of the current coding block is the left        boundary of the tile and loop_filter_across_tiles_enabled_flag        is equal to 0.    -   The left boundary of the current coding block is the left        boundary of the tile group and        tile_group_loop_filter_across_tile_groups_enabled_flag is equal        to 0.    -   The left boundary of the current coding block is the left        boundary of the current tile group and all the following        conditions are satisfied:    -   gdr_enabled_flag is equal to 1    -   loop_filter_across_refreshed_region_enabled_flag is equal to 0.    -   the tile group that shared the boundary with the left boundary        of the current tile group is present and the value of its        refreshed_region_flag is different from the value of        refreshed_region_flag of the current tile group.    -   Otherwise if edgeType is equal to EDGE_HOR and one or more of        the following conditions are true, the variable filterEdgeFlag        is set equal to 0:    -   The top boundary of the current luma coding block is the top        boundary of the picture.    -   The top boundary of the current coding block is the top boundary        of the tile and loop_filter_across_tiles_enabled_flag is equal        to 0.    -   The top boundary of the current coding block is the top boundary        of the tile group and        tile_group_loop_filter_across_tile_groups_enabled_flag is equal        to 0.    -   The top boundary of the current coding block is the top boundary        of the current tile group and all the following conditions are        satisfied.    -   gdr_enabled_flag is equal to 1    -   loop_filter_across_refreshed_region_enabled_flag is equal to 0.    -   the tile group that shared the boundary with the top boundary of        the current tile group is present and the value of its        refreshed_region_flag is different from the value of        refreshed_region_flag of the current tile group.    -   Otherwise, filterEdgeFlag is set equal to 1.

Adapt syntax once tiles are integrated.

3. All elements of the two-dimensional (nCbW)×(nCbH) array edgeFlags areinitialized to be equal to zero.

CTB modification process for SAO is discussed.

. . .

For all sample locations (xS_(i), yS_(j)) and (xY_(i), yY_(j)) with i=0. . . nCtbSw−1 and j=0 . . . nCtbSh−1, depending on the values ofpcm_loop_filter_disabled_flag, pcm_flag[xY_(i)][yY_(j)] andcu_transquant_bypass_flag of the coding unit which includes the codingblock covering recPicture[xSi][ySj], the following applies:

-   -   . . .

Modify highlighted sections pending on future decisiontransform/quantization bypass.

-   -   Otherwise, if SaoTypeIdx[cIdx][rx][ry] is equal to 2, the        following ordered steps apply:

1. The values of hPos[k] and vPos[k] for k=0 . . . 1 are specified inTable 8-18 based on SaoEoClass[cIdx][rx][ry].

2. The variable edgeIdx is derived as follows:

-   -   The modified sample locations (xS_(ik′), yS_(jk′)) and        (xY_(ik′), yY_(jk′)) are derived as follows:        (xS _(ik′) ,yS _(jk′))=(xS _(i) +hPos[k],yS _(j)        +vPos[k])  (8-1128)        (xY _(ik′) ,yY _(jk′))=(cIdx==0)?(xS _(ik′) ,yS _(jk′)):(xS        _(ik′)*SubWidthC,yS _(jk′)*SubHeightC)  (8-1129)    -   If one or more of the following conditions for all sample        locations (xS_(ik′), yS_(jk′)) and (xY_(ik′), yY_(jk′)) with k=0        . . . 1 are true, edgeIdx is set equal to 0:    -   The sample at location (xS_(ik′), yS_(jk′)) is outside the        picture boundaries.    -   gdr_enabled_flag is equal o 1,        loop_filter_across_refreshed_region_enabled_flag is equal to 0,        refreshed_region_flag of the current tile group is equal to 1,        and refreshed_region_flag of the tile group containing the        sample at location (xS_(ik′), yS_(jk′)) is equal to 0.    -   The sample at location (xS_(ik′), yS_(jk′)) belongs to a        different tile group and one of the following two conditions is        true:    -   MinTbAddrZs[xY_(ik′)>>MinTbLog 2SizeY][yY_(jk′)>>MinTbLog        2SizeY] is less than MinTbAddrZs[xY_(i)>>MinTbLog        2SizeY][yY_(j)>>MinTbLog 2SizeY] and        tile_group_loop_filter_across_tile_groups_enabled_flag in the        tile group which the sample recPicture[xS_(i)][yS_(j)] belongs        to is equal to 0.    -   MinTbAddrZs[xY_(i)>>MinTbLog 2SizeY][yY_(j)>>MinTbLog 2SizeY] is        less than MinTbAddrZs[xY_(ik′)>>MinTbLog        2SizeY][yY_(jk′)>>MinTbLog 2SizeY] and        tile_group_loop_filter_across_tile_groups_enabled_flag in the        tile group which the sample recPicture[xS_(ik′)][yS_(jk′)]        belongs to is equal to 0.    -   loop_filter_across_tiles_enabled_flag is equal to 0 and the        sample at location (xS_(ik′), yS_(jk′)) belongs to a different        tile.

Modify highlighted sections when tiles without tile groups areincorporated

-   -   Otherwise, edgeIdx is derived as follows:    -   The following applies:        edgeIdx=2+Sign(recPicture[xS _(i) ][yS _(j)]−recPicture[xS _(i)        +hPos[0]][yS _(j) +vPos[0]])+Sign(recPicture[xS _(i) ][yS        _(j)]−recPicture[xS _(i) +hPos[1]][yS _(j) +vPos[1]])   (8-1130)    -   When edgeIdx is equal to 0, 1, or 2, edgeIdx is modified as        follows:        edgeIdx=(edgeIdx==2)?0:(edgeIdx+1)  (8-1131)

3. The modified picture sample array saoPicture[xS_(i)][yS_(j)] isderived as follows:saoPicture[xS _(i) ][yS _(j)]=Clip3(0,(1<<bitDepth)−1,recPicture[xS _(i)][yS _(j)]+SaoOffsetVal[cIdx][rx][ry][edgeIdx])  (8-1132)

Coding tree block filtering process for luma samples for ALF isdiscussed.

. . .

For the derivation of the filtered reconstructed luma samplesalfPictureL[x][y], each reconstructed luma sample inside the currentluma coding tree block recPictureL[x][y] is filtered as follows with x,y=0 . . . CtbSizeY−1:

-   -   . . .

The locations (h_(x), v_(y)) for each of the corresponding luma samples(x, y) inside the given array recPicture of luma samples are derived asfollows:

-   -   If gdr_enabled_flag is equal to 1,        loop_filter_across_refreshed_region_enabled_flag is equal to 0,        refreshed_region_flag of the tile group tgA containing the luma        sample at location (x, y) is equal to 1, the following applies:    -   If the location (h_(x), v_(y)) is located in another tile group        tgB and the refreshed_region_flag of tgB is equal to 0, the        variables leftBoundary, rightBoundary, topBoundary, and        botBoundary are set equal to TGRefreshedLeftBoundary,        TGRefreshedRightBoundary, TGRefreshedTopBoundary, and        TGRefreshedBotBoundary, respectively.    -   Otherwise, the variables leftBoundary, rightBoundary,        topBoundary, and botBoundary are set equal to        PicRefreshedLeftBoundaryPos, PicRefreshedRightBoundaryPos,        PicRefreshedTopBoundaryPos, and PicRefreshedBotBoundaryPos,        respectively.        h _(x)=Clip3(leftBoundary,rightBoundary,xCtb+x)  (8-1140)        v _(y)=Clip3(topBoundary,botBoundary,yCtb+y)  (8-1141)    -   Otherwise, the following applies:        h _(x)=Clip3(0,pic_width_in_luma_samples−1,xCtb+x)  (8-1140)        v _(y)=Clip3(0,pic_height_in_luma_samples−1,yCtb+y)  (8-1141)        . . .

The derivation process for ALF transpose and filter index for lumasamples is discussed.

. . .

The locations (h_(x), v_(y)) for each of the corresponding luma samples(x, y) inside the given array recPicture of luma samples are derived asfollows:

-   -   If gdr_enabled_flag is equal to 1,        loop_filter_across_refreshed_region_enabled_flag is equal to 0,        refreshed_region_flag of the tile group tgA containing the luma        sample at location (x, y) is equal to 1, the following applies:    -   If the location (h_(x), v_(y)) is located in another tile group        tgB and the refreshed_region_flag of tgB is equal to 0, the        variables leftBoundary, rightBoundary, topBoundary, and        botBoundary are set equal to TGRefreshedLeftBoundary,        TGRefreshedRightBoundary, TGRefreshedTopBoundary, and        TGRefreshedBotBoundary, respectively.    -   Otherwise, the variables leftBoundary, rightBoundary,        topBoundary, and botBoundary are set equal to        PicRefreshedLeftBoundaryPos, PicRefreshedRightBoundaryPos,        PicRefreshedTopBoundaryPos, and PicRefreshedBotBoundaryPos,        respectively.        h _(x)=Clip3(leftBoundary,rightBoundary,x)  (8-1140)        v _(y)=Clip3(topBoundary,botBoundary,y)  (8-1141)    -   Otherwise, the following applies:        h _(x)=Clip3(0,pic_width_in_luma_samples−1,x)  (8-1145)        v _(y)=Clip3(0,pic_height_in_luma_samples−1,y)  (8-1146)

Coding tree block filtering process for chroma samples is discussed.

. . .

For the derivation of the filtered reconstructed chroma samplesalfPicture[x][y], each reconstructed chroma sample inside the currentchroma coding tree block recPicture[x][y] is filtered as follows with x,y=0 . . . ctbSizeC−1:

-   -   The locations (h_(x), v_(y)) for each of the corresponding        chroma samples (x, y) inside the given array recPicture of        chroma samples are derived as follows:    -   If gdr_enabled_flag is equal to 1,        loop_filter_across_refreshed_region_enabled_flag is equal to 0,        refreshed_region_flag of the tile group tgA containing the luma        sample at location (x, y) is equal to 1, the following applies:    -   If the location (h_(x), v_(y)) is located in another tile group        tgB and the refreshed_region_flag of tgB is equal to 0, the        variables leftBoundary, rightBoundary, topBoundary, and        botBoundary are set equal to TGRefreshedLeftBoundary,        TGRefreshedRightBoundary, TGRefreshedTopBoundary, and        TGRefreshedBotBoundary, respectively.    -   Otherwise, the variables leftBoundary, rightBoundary,        topBoundary, and botBoundary are set equal to        PicRefreshedLeftBoundaryPos, PicRefreshedRightBoundaryPos,        PicRefreshedTopBoundaryPos, and PicRefreshedBotBoundaryPos,        respectively.        h        _(x)==Clip3(leftBoundary/SubWidthC,rightBoundary/SubWidthC,xCtbC+x)          (8-1140)        v        _(y)=Clip3(topBoundary/SubWidthC,botBoundary/SubWidthC,yCtbC+y)          (8-1141)    -   Otherwise, the following applies:        h        _(x)=Clip3(0,pic_width_in_luma_samples/SubWidthC−1,xCtbC+x)  (8-1177)        v        _(y)=Clip3(0,pic_height_in_luma_samples/SubHeightC−1,yCtbC+y)  (8-1178)

FIG. 7 illustrates a video bitstream 750 configured to implement agradual decoding refresh (GDR) technique 700 according to an embodimentof the disclosure. The GDR technique 700 may be similar to the GDRtechnique 500 of FIG. 5 . As used herein the video bitstream 750 mayalso be referred to as a coded video bitstream, a bitstream, orvariations thereof. As shown in FIG. 7 , the bitstream 750 comprises asequence parameter set (SPS) 752, a picture parameter set (PPS) 754, aslice header 756, and image data 758.

The SPS 752 contains data that is common to all the pictures in asequence of pictures (SOP). In contrast, the PPS 754 contains data thatis common to the entire picture. The slice header 756 containsinformation about the current slice such as, for example, the slicetype, which of the reference pictures will be used, and so on. The SPS752 and the PPS 754 may be generically referred to as a parameter set.The SPS 752, the PPS 754, and the slice header 756 are types of NetworkAbstraction Layer (NAL) units. A NAL unit is a syntax structurecontaining an indication of the type of data to follow (e.g., codedvideo data). NAL units are classified into video coding layer (VCL) andnon-VCL NAL units. The VCL NAL units contain the data that representsthe values of the samples in the video pictures, and the non-VCL NALunits contain any associated additional information such as parametersets (important header data that can apply to a large number of VCL NALunits) and supplemental enhancement information (timing information andother supplemental data that may enhance usability of the decoded videosignal but are not necessary for decoding the values of the samples inthe video pictures). Those skilled in the art will appreciate that thebitstream 750 may contain other parameters and information in practicalapplications.

The image data 758 of FIG. 7 comprises data associated with the imagesor video being encoded or decoded. The image data 758 may be simplyreferred to as the payload or data being carried in the bitstream 750.In an embodiment, the image data 758 comprises the CVS 708 (or CLVS)containing a GDR picture 702, one or more trailing pictures 704, and arecovery point picture 706. The image data 758 may also contain apicture 760 disposed in a NAL unit 730 with an end of sequence (EOS) NALunit type (EOS_NUT). The NAL unit 730 with the EOS_NUT is the NAL unitcontaining slices from the last picture in the CVS. The NAL unit 730containing the picture 760 may be referred to herein as an EOS NAL unit.As shown, the picture 760 in the NAL unit 730 preceeds the GDR picture702 and is disposed in a different CVS than the GDR picture 702. In anembodiment, the picture 760 is in a CVS immediately preceeding the CVS708.

In an embodiment, the GDR picture 702, the trailing pictures 704, andthe recovery point picture 706 may define a GDR period in the CVS 708.In an embodiment, a decoding order begins with the GDR picture 702,continues with the trailing pictures 704, and then proceeds to therecovery point picture 706. In an embodiment, the trailing pictures 704may be considered a form of GDR picture since they precede the recoverypoint picture 706 in the GDR period.

When the GDR picture 702 is the initial picture in decoding order or isthe initial picture following the EOS NAL unit 730 in the decodingorder, the value of a flag is set to one (1) to prevent the GDR picture702 and any trailing pictures 704 between the GDR picture 702 and therecovery point picture 706 in output order (e.g., presentation order410) from being output. Otherwise, the flag is set to zero (0). That is,the flag is set to zero when the GDR picture is not the initial picturein the decoding order and is not the initial picture following the EOSNAL unit in the decoding order. In an embodiment, the flag is designatedNoOutPutBeforeRecoveryFlag. In an embodiment, only the GDR picture 702is prevented from being output when the flag is set to one.

The CVS 708 is a coded video sequence for every coded layer videosequence (CLVS) in the video bitstream 750. Notably, the CVS and theCLVS are the same when the video bitstream 750 includes a single layer.The CVS and the CLVS are only different when the video bitstream 750includes multiple layers. A layer is set of VCL NAL units that all havea particular value (e.g., the same layer ID) and the associated non-VCLNAL units, or one of a set of syntactical structures having ahierarchical relationship

As shown in FIG. 7 , the GDR technique 700 or principle works over aseries of pictures starting with the GDR picture 702 and ending with therecovery point picture 706. The GDR technique 700, the GDR picture 702,the trailing pictures 704, and the recovery point picture 706 aresimilar to the GDR technique 500, the GDR picture 502, the trailingpictures 504, and the recovery point picture 506 of FIG. 5 . Therefore,for the sake of brevity, the manner in which the GDR technique 700 isimplemented is not repeated with regard to FIG. 7 .

As shown in FIG. 7 , the GDR picture 702, the trailing pictures 704, andthe recovery point picture 706 in the CVS 708 are each contained withintheir own VCL NAL unit 730. The set of VCL NAL units 730 in the CVS 708may be referred to as an access unit.

The NAL unit 730 containing the GDR picture 702 in the CVS 708 has a GDRNAL unit type (GDR_NUT). That is, in an embodiment the NAL unit 730containing the GDR picture 702 in the CVS 708 has its own unique NALunit type relative to the trailing pictures 704 and the recovery pointpicture 706. In an embodiment, the GDR_NUT permits the bitstream 750 tobegin with the GDR picture 702 instead of the bitstream 750 having tobegin with an IRAP picture. Designating the VCL NAL unit 730 of the GDRpicture 702 as a GDR_NUT may indicate to, for example, a decoder thatthe initial VCL NAL unit 730 in the CVS 708 contains the GDR picture702.

In an embodiment, the GDR picture 702 is the initial picture in the CVS708. In an embodiment, the GDR picture 702 is the initial picture in theGDR period. In an embodiment, the GDR picture 702 has a temporalidentifier (ID) equal to zero. The temporal ID is value or number thatidentifies the position or order of a picture relative to otherpictures. In an embodiment, an access unit containing the VCL NAL unit730 having the GDR_NUT is designated a GDR access unit. In anembodiment, the GDR picture 702 is a coded slice of another (e.g.,larger) GDR picture. That is, the GDR picture 702 may be a portion of alarger GDR picture.

FIG. 8 is an embodiment of a method 800 of decoding a coded videobitstream implemented by a video decoder (e.g., video decoder 30). Themethod 800 may be performed after the decoded bitstream has beendirectly or indirectly received from a video encoder (e.g., videoencoder 20). The method 800 improves the decoding process because themethod allows progressive intra refresh to enable random access withouthaving to use an IRAP picture. The value of a flag is set to a value toprevent a GDR picture and any trailing pictures between the GDR pictureand a recovery point picture in output order from being output when theGDR picture is the initial picture in decoding order or the initialpicture following the EOS NAL unit in the decoding order. Setting theflag in this manner prevents potentially dirty data from being output toa display. Therefore, as a practical matter, the performance of a codecis improved, which leads to a better user experience.

In block 802, the video decoder determines whether a GDR picture (e.g.,the GDR picture 702) in the coded video bitstream (e.g., bitstream 750)is an initial picture in decoding order or is the initial picturefollowing an EOS NAL unit in the decoding order.

In an embodiment, the GDR picture (e.g., GDR picture 702) is the initialpicture in a CVS of the coded video bitstream (coded video bitstream750). In an embodiment, the GDR picture is the initial picture in alayer of the coded video bitstream. In an embodiment, the layer is aCLVS of a CVS of the coded video bitstream.

In block 804, the video decoder sets a flag to a value to prevent theGDR picture from being output when the GDR picture is the initialpicture in decoding order or the initial picture following the EOS NALunit in the decoding order. In an embodiment, the video decoder sets aflag to a value to prevent the GDR picture and any trailing picturesbetween the GDR picture and a recovery point picture in output orderfrom being output when the GDR picture is the initial picture indecoding order or the initial picture following the EOS NAL unit in thedecoding order. In an embodiment, the value of the flag is set to one(1) to prevent the current picture and any trailing pictures between theGDR picture and a recovery point picture in output order from beingoutput when the GDR picture is the initial picture in the decoding orderor the initial picture following the EOS NAL unit in the decoding order.In an embodiment, the value of the flag is set to zero (0) when the GDRpicture is not the initial picture in the decoding order and is not theinitial picture following the EOS NAL unit in the decoding order.

In block 806, the video decoder decodes the GDR picture. The trailingpictures and the recovery point pictures are then decoded in turn. Inblock 808, the video decoder stores the GDR picture in a decoded picturebuffer (DPB). In an embodiment, when the GDR picture has not beenrestricted from being output by the setting of the flag, an imagegenerated based on the GDR picture may be displayed for a user of anelectronic device (e.g., a smart phone, tablet, laptop, personalcomputer, etc.).

FIG. 9 is a schematic diagram of a video coding device 900 (e.g., avideo encoder 20 or a video decoder 30) according to an embodiment ofthe disclosure. The video coding device 900 is suitable for implementingthe disclosed embodiments as described herein. The video coding device900 comprises ingress ports 910 and receiver units (Rx) 920 forreceiving data; a processor, logic unit, or central processing unit(CPU) 930 to process the data; transmitter units (Tx) 940 and egressports 950 for transmitting the data; and a memory 960 for storing thedata. The video coding device 900 may also compriseoptical-to-electrical (OE) components and electrical-to-optical (EO)components coupled to the ingress ports 910, the receiver units 920, thetransmitter units 940, and the egress ports 950 for egress or ingress ofoptical or electrical signals.

The processor 930 is implemented by hardware and software. The processor930 may be implemented as one or more CPU chips, cores (e.g., as amulti-core processor), field-programmable gate arrays (FPGAs),application specific integrated circuits (ASICs), and digital signalprocessors (DSPs). The processor 930 is in communication with theingress ports 910, receiver units 920, transmitter units 940, egressports 950, and memory 960. The processor 930 comprises a coding module970. The coding module 970 implements the disclosed embodimentsdescribed above. For instance, the coding module 970 implements,processes, prepares, or provides the various codec functions. Theinclusion of the coding module 970 therefore provides a substantialimprovement to the functionality of the video coding device 900 andeffects a transformation of the video coding device 900 to a differentstate. Alternatively, the coding module 970 is implemented asinstructions stored in the memory 960 and executed by the processor 930.

The video coding device 900 may also include input and/or output (I/O)devices 980 for communicating data to and from a user. The I/O devices980 may include output devices such as a display for displaying videodata, speakers for outputting audio data, etc. The I/O devices 980 mayalso include input devices, such as a keyboard, mouse, trackball, etc.,and/or corresponding interfaces for interacting with such outputdevices.

The memory 960 comprises one or more disks, tape drives, and solid-statedrives and may be used as an over-flow data storage device, to storeprograms when such programs are selected for execution, and to storeinstructions and data that are read during program execution. The memory960 may be volatile and/or non-volatile and may be read-only memory(ROM), random access memory (RAM), ternary content-addressable memory(TCAM), and/or static random-access memory (SRAM).

FIG. 10 is a schematic diagram of an embodiment of a means for coding1000. In an embodiment, the means for coding 1000 is implemented in avideo coding device 1002 (e.g., a video encoder 20 or a video decoder30). The video coding device 1002 includes receiving means 1001. Thereceiving means 1001 is configured to receive a picture to encode or toreceive a bitstream to decode. The video coding device 1002 includestransmission means 1007 coupled to the receiving means 1001. Thetransmission means 1007 is configured to transmit the bitstream to adecoder or to transmit a decoded image to a display means (e.g., one ofthe I/O devices 1080).

The video coding device 1002 includes a storage means 1003. The storagemeans 1003 is coupled to at least one of the receiving means 1001 or thetransmission means 1007. The storage means 1003 is configured to storeinstructions. The video coding device 1002 also includes processingmeans 1005. The processing means 1005 is coupled to the storage means1003. The processing means 1005 is configured to execute theinstructions stored in the storage means 1003 to perform the methodsdisclosed herein.

It should also be understood that the steps of the exemplary methods setforth herein are not necessarily required to be performed in the orderdescribed, and the order of the steps of such methods should beunderstood to be merely exemplary. Likewise, additional steps may beincluded in such methods, and certain steps may be omitted or combined,in methods consistent with various embodiments of the presentdisclosure.

While several embodiments have been provided in the present disclosure,it should be understood that the disclosed systems and methods might beembodied in many other specific forms without departing from the spiritor scope of the present disclosure. The present examples are to beconsidered as illustrative and not restrictive, and the intention is notto be limited to the details given herein. For example, the variouselements or components may be combined or integrated in another systemor certain features may be omitted, or not implemented.

In addition, techniques, systems, subsystems, and methods described andillustrated in the various embodiments as discrete or separate may becombined or integrated with other systems, modules, techniques, ormethods without departing from the scope of the present disclosure.Other items shown or discussed as coupled or directly coupled orcommunicating with each other may be indirectly coupled or communicatingthrough some interface, device, or intermediate component whetherelectrically, mechanically, or otherwise. Other examples of changes,substitutions, and alterations are ascertainable by one skilled in theart and could be made without departing from the spirit and scopedisclosed herein.

What is claimed is:
 1. A method implemented by a decoder, the methodcomprising: determining whether a gradual decoding refresh (GDR) picturein a bitstream is an initial picture in decoding order or is the initialpicture following an end of sequence (EOS) network abstraction layer(NAL) unit in the decoding order; setting a flag to a value to preventthe GDR picture from being output when the GDR picture is the initialpicture in the decoding order or the initial picture following the EOSNAL unit in the decoding order, wherein the value of the flag is set toone (1) to prevent the GDR picture from being output when the GDRpicture is the initial picture in the decoding order or the initialpicture following the EOS NAL unit in the decoding order; decoding theGDR picture; and storing the GDR picture in a decoded picture buffer(DPB).
 2. The method of claim 1, wherein the flag is set to the value toprevent any trailing pictures between the GDR picture and a recoverypoint picture in output order from being output when the GDR picture isthe initial picture in the decoding order or the initial picturefollowing the EOS NAL unit in the decoding order.
 3. The method of claim1, wherein the GDR picture is the initial picture in a coded videosequence (CVS) of the bitstream.
 4. The method of claim 1, wherein theGDR picture is the initial picture in a layer of the bitstream.
 5. Themethod of claim 4, wherein the layer includes a coded layer videosequence (CLVS) of a coded video sequence (CVS) of the bitstream.
 6. Themethod of claim 1, wherein the value of the flag is set to zero (0) whenthe GDR picture is not the initial picture in the decoding order and isnot the initial picture following the EOS NAL unit in the decodingorder.
 7. A non-transitory computer readable medium comprising acomputer program product for use by a decoding device, the computerprogram product comprising computer executable instructions stored onthe non-transitory computer readable medium such that when executed by aprocessor cause the decoding device to: receive a coded video bitstream;determine whether a gradual decoding refresh (GDR) picture in the codedvideo bitstream is an initial picture in decoding order or is theinitial picture following an end of sequence (EOS) network abstractionlayer (NAL) unit in the decoding order; set a flag to a value to preventthe GDR picture from being output when the GDR picture is the initialpicture in the decoding order or the initial picture following the EOSNAL unit in the decoding order, wherein the value of the flag is set toone (1) to prevent the GDR picture from being output when the GDRpicture is the initial picture in the decoding order or the initialpicture following the EOS NAL unit in the decoding order; decode the GDRpicture; and store the GDR picture in a decoded picture buffer (DPB). 8.The non-transitory computer readable medium of claim 7, wherein the flagis set to the value to prevent the GDR picture and any trailing picturesbetween the GDR picture and a recovery point picture in output orderfrom being output when the GDR picture is the initial picture in thedecoding order or the initial picture following the EOS NAL unit in thedecoding order.
 9. The non-transitory computer readable medium of claim7, wherein the GDR picture is the initial picture in a coded videosequence (CVS) of the coded video bitstream, or wherein the GDR pictureis the initial picture in a layer of the coded video bitstream.
 10. Thenon-transitory computer readable medium of claim 9, wherein the layer isa coded layer video sequence (CLVS) of a coded video sequence (CVS) ofthe coded video bitstream.
 11. The non-transitory computer readablemedium of claim 7, wherein the value of the flag is set to zero (0) whenthe GDR picture is not the initial picture in the decoding order and isnot the initial picture following the EOS NAL unit in the decodingorder.
 12. A decoder comprising: a processor configured to: determinewhether a gradual decoding refresh (GDR) picture in a bitstream is aninitial picture in decoding order or is the initial picture following anend of sequence (EOS) network abstraction layer (NAL) unit in thedecoding order; set a flag to a value to prevent the GDR picture frombeing output when the GDR picture is the initial picture in the decodingorder or the initial picture following the EOS NAL unit in the decodingorder, wherein the value of the flag is set to one (1) to prevent theGDR picture from being output when the GDR picture is the initialpicture in the decoding order or the initial picture following the EOSNAL unit in the decoding order; and decode the GDR picture; and a memorycoupled to the processor and configured to store the GDR picture in adecoded picture buffer (DPB).
 13. The decoder of claim 12, wherein theflag is set to the value to prevent any trailing pictures between theGDR picture and a recovery point picture in output order from beingoutput when the GDR picture is the initial picture in the decoding orderor the initial picture following the EOS NAL unit in the decoding order.14. The decoder of claim 12, wherein the GDR picture is the initialpicture in a coded video sequence (CVS) of the bitstream.
 15. Thedecoder of claim 12, wherein the GDR picture is the initial picture in alayer of the bitstream.
 16. The decoder of claim 15, wherein the layerincludes a coded layer video sequence (CLVS) of a coded video sequence(CVS) of the bitstream.
 17. The decoder of claim 12, wherein the valueof the flag is set to zero (0) when the GDR picture is not the initialpicture in the decoding order and is not the initial picture followingthe EOS NAL unit in the decoding order.